Trading the Markets With FOSS Software? 417
Robert writes "Along with many other techies, I share an interest in the world of finance (bubble-era stock options pulled me in). Unfortunately, as someone with a strong preference for GNU/Linux as my operating system of choice, I have found that software in this area seems quite sparse. For awhile I have made do with Python, R, Gnumeric, Gnucash and a telephone, along with some small utilities I have written myself. What I would like to know is: what FOSS software do you use for financial analysis, trading, system development, and testing in a Un*x environment? Are there programs you would like to see written or ported? Do any brokerages, data providers, or other services provide good support for we the few? And finally, what commercial entities do you know of that are using FOSS software in their operation?"
Let me save you the trouble (Score:3, Informative)
There is none. You will get lots of recommendations to "hack it" with a hodgepodge of crap like the one you seem to be using right now, but that gets you nowhere.
There is no quality trading/management trading software on any OS other than Windows that bears even a passing mention.
Posting AC because people around here tend to get tender and defensive when someone dares suggest that their techno-religious experience is not absolutely perfect for everyone in the planet. Email and surfing the web? Linux is great. This? Don't waste your time and just stick with Windows.
(cue twitter to tell me [slashdot.org] how I "hate freedom")
EclipseTrader (Score:5, Informative)
EclipseTrader [sourceforge.net] is probably the most advanced open source trading program. It interfaces with some trading platforms and intra-day data feeds. It has several hundred technical indicators. It also is very expandable and easy to write modules for (in Java). I wrote some technical analysis modules for the back-testing system and was fairly impressed with how well it worked as it is based on the very solid OSGI/Eclipse model. I'd say it actually competes fairly well with some of the proprietary trading platforms I have used, especially if you are a Java coder and want to add modules to it to aid in implementing your particular trading style.
Tons of stuff out there: (Score:5, Informative)
I understand that the submitter doesn't know what is out there, but if you ever have a trading account you're likely to have API access to your broker's systems. I recommend Interactive Brokers as their TWS software has a lot of different language bindings.
In finance you don't look for "FOSS" tools, you go to your broker, get API access, and write them yourself.
Re:EclipseTrader (Score:4, Informative)
Last time I tried EcT it consistently crashed on my Gentoo machine without warning, and it consumes unholy amounts of memory. I never trusted it enough to actually use it as a trading platform. For keeping an eye on your accounts and whatnot it's fine, but unless they've made heroic strides in the past five months, it's probably still unstable. And for that particular purpose I can use the excellent only Google interface.
The community around it is also rather thin, and (IMO) actually downright hostile to recommendations and bug reports.
Frankly, I'd rather trust being able to burst a sell order to an application that a company has spent a few millions developing and actively supports than to a "community project". So far I haven't had any problems with the former. Sure, I have to dual boot into Windows (no WINE for the one I use, at least not yet), but this is my livelyhood we're talking about.
Technical analysis SW for Linux: Qtstalker (Score:1, Informative)
There's quite a decent (altough not very polished) charting + technical analysis program for Linux called Qtstalker: http://qtstalker.sourceforge.net/ . You can import stock quotes from Yahoo finance or other sources automatically, or commodity/stock data from .csv files, which most data suppliers offer for download.
Interactive Brokers (Score:1, Informative)
Interactive Brokers has a Java based client which runs nicely in linux. It provides an API for C++ and Java to execute trades and get data. OpenTick provides historical data and realtime data via an API in Java or C++.
It is not open source, but it does run in Linux.
Interactive Brokers's Java trading platform (TWS) (Score:5, Informative)
Interactive Brokers [interactivebrokers.com] has a Java-based
Trader Workstation [interactivebrokers.com] ("TWS") and they explicitly support Linux. They offer almost anything you can get anywhere, including mutual funds, stocks, options, futures (commodity & financial), currency, and foreign stocks. Commissions are 10x lower than Charles Schwab if you trade often (if you don't then a minimum monthly commission kicks in).
TWS is a large, cumbersome Java applet, but it works tolerably well on a fast machine (and there's not much alternative on Linux)
One annoyance is that they only support jdk 1.5.0_x (not the current 1.6.x), I think because of some concurrency bugs in their code (they claim the newer Java is buggy). However TWS generally does work with the latest jdk, but they won't support it.
IB's telephone support is sometimes rude, the opposite of "hand holding". I guess they have only a few over-worked support people to keep costs down. Also, they only provide on-line statements and never send physical mail except for annual 1099 tax forms. So, be sure your spouse/executor knows you have an IB account, because if you die there will be no monthly statements to clue them!
In summary, IB is good, despite their warts. If you trade a few times a month or more, it's worth the hassles.
Re:Hmmm (Score:4, Informative)
no collateral or responsibility required!
Ummm...please check your facts. AIG's assets are collateral and the loan is 11%+. I pay a lot less than than that on the equity lines and mortgages I have with my real estate investments. Sure, it's a bail out, but AIG's going to spend some big bucks paying it back or lose big time.
Financial Literacy is most people's problem, thus their finances are a problem.
TD Ameritrade has API (Score:2, Informative)
I know I don't actually mention any FOSS solutions but as a mostly FOSS user this is how I trade.
Umm... Firefox. Perhaps with Flash. (Score:3, Informative)
Being most of this stuff needs a network connection anyways I am sure you will find particular websites that will do the trick. Why do you want to find and download an app, run an update every time the regulations change. Most of the apps for this stuff are so old and usually try to get people to use the web anyways. Don't look for an app when the Web can do the work just as well if not better.
Well, since I develop trading systems on FOSS (Score:5, Informative)
That is the trading system I've spent several years working on is built entirely using Open Source tools and libraries. The system itself is not currently open, but that is a possibility we here certainly look favorably at.
As far as actual entire free trading systems, there is JavaTraders@googlegroups.com which is a good place to start. Also check out the quickfixj.org site, you will find some things there. There is also an Eclipse plugin which provides some level of GUI.
Frankly we didn't any of the code in any of those projects (although we do use ta-lib). But as I say, you can do a lot with ActiveMQ, any good open source RDBMS (PostgreSQL,MySQL) and your Enterprise Java framework bits of choice.
Basically if I were you I'd pick one of the java based projects that is kicking and does roughly what you want, the way you want to do it. For simple basic trading of one or two instrument classes you can probably put together something pretty workable.
Re:Hmmm (Score:4, Informative)
Resources of Linux/Java trading software... (Score:3, Informative)
I have not seen any open source projects that you can use as a platform for building a trading system.
I have built an intraday (within one day) trading system in Java. I'm afraid that this system is not open source either. This system runs one or more models that look for intraday trading signals. The Java software submits buy and sell orders. It is multi-threaded and runs one thread per stock. I have been very happy with the software performance. A long running "server" like this seems to benefit from Sun's HotSpot compiler. The system is web services based (e.g., it runs on Tomcat).
I used Interactive Brokers [interactivebrokers.com] for my market data and order infrastructure. I was concerned about the quality of the Interactive Brokers tick data (the trade by trade data). Interactive Brokers consolidates their tick data feed so you get a consolidated tick about ever 250 msec. For my system this has been adequate. If you want to run on Linux or use Java there are few inexpensive options for real time data feeds. Information may way to be free, but market data is expensive.
I have some web pages on the alternatives that I explored as platforms for a Java/Linux based trading system. These notes can be found on my web page Software for Constructing a Market Trading System [bearcave.com]
Re:Let me save you the trouble (Score:5, Informative)
Areed. I spent 3 months trying to find a good solution. If you've got buckets of money, NxCore ( http://www.nanex.net/NxCore/NxCore.htm [nanex.net] - prices start at $500/month) and any of the brokers that support a FIX API (for which you can expect to pay a hefty fee, too; Interactive Brokers (IB - http://www.interactivebrokers.com/ [interactivebrokers.com] ) for example charge a one time $500 fee, OANDA ( http://www.oanda.com/ [oanda.com] ) charge $600 for the first two months then an ongoing subscription fee if you trade $12mil/month or something).
For those people not wanting to pour money into it, as good as you can get is Interactive Broker's Trader Workstation (TWS), and JBookTrader (http://code.google.com/p/jbooktrader/) or a custom trading platform that talks to their API. TWS is a pain that lacks automated login (for security reasons) and auto-exits every 24 hours (for... err... security reasons?), but it gets the job done. Data feed can be an issue still, though; IB offer up to 100 symbols at a time, and a basic historical data service, but some people dislike the fact they drop price ticks during busy market times (over 10 prices per second) and the historical data service is paced so you can only do a limited of number of requests (about one every ten seconds I believe).
In short though, AC is right; use Windows, it may well be less painful. Really.
Re:You are agreeing! (Score:3, Informative)
Because the normal regulations were absent! And why were they absent? They were changed... in Washington!
Those would be the "normal regulations" that brought us Enron, Worldcom, Global Crossing and friends? The Bush Administration has nothing to be proud of, but the concept of corporate fraud was not invented on their watch.
Predicted in 99 (Score:5, Informative)
Interesting speech from Senator Dorgan when the bill, Financial Services Modernization Act, was being discussed '99. Those who don't know history...
"I remember a couple of circumstances that existed more recently. I was not around during the bank failures of the 1930s. I was not around for the debate that persuaded a Congress to enact Glass-Steagall and a range of other protections. But I was here when, in the early 1980s, it was decided that we should expand the opportunities for savings and loans to do certain things. And they began to broker deposits and they took off. They would take a sleepy little savings and loan in some town, and they would take off like a Roman candle. Pretty soon they would have a multibillion-dollar organization, and they would decide they would use that organization to park junk bonds in. We had a savings and loan out in California that had over 50 percent of its assets in risky junk bonds.
Let me describe the ultimate perversion, the hood ornament on stupidity. The U.S. Government owned nonperforming junk bonds in the Taj Mahal Casino. Let me say that again. The U.S. Government ended up owning nonperforming junk bonds in the Taj Mahal Casino in Atlantic City. How did that happen? The savings and loans were able to buy junk bonds. The savings and loans went belly up. The junk bonds were not performing. And the U.S. Government ended up with those junk bonds.
Was that a perversion? Of course it was. But it is an example of what has happened when we decide, under a term called modernization, to forget the lessons of the past, to forget there are certain things that are inherently risky, and they ought not be fused or merged with the enterprise of banking that requires the perception and, of course, the reality--but especially the perception--of safety and soundness.
Last year, we had a failure of a firm called LTCM, Long-Term Capital Management. It was an organization run by some of the smartest people in the world, I guess, in the area of finance. They had Nobel laureates helping run this place. They had some of the smartest people on Wall Street. They put together a lot of money. They had this hedge fund, unregulated hedge fund. They had invested more than $1 trillion in derivatives in this fund--more than $1 trillion in derivatives value.
Then, with all of the smartest folks around, and all this money, and an enormous amount of leverage, when it looked as if this firm was going to go belly up, just flat out broke, guess what happened. On a Sunday, Mr. Greenspan and the Federal Reserve Board decided to convene a meeting of corresponding banks and others who had an interest in this, saying: You have to save Long-Term Capital Management. You have to save this hedge firm. If you don't, there will be catastrophic results in the economy. The hit will be too big.
You have this unregulated risky activity out there in the economy, and you have one firm that has $1 trillion in derivative values and enormous risk, and, with all their brains, it doesn't work. They are going to go belly up. Who bears the burden of that? The Federal Government, the Federal Reserve Board.
We have the GAO doing an investigation to find out the circumstances of all that. I am very interested in this no-fault capitalism that exists with respect to Long-Term Capital Management. Who decides what kind of capitalism is no-fault capitalism? And when and how and is there a conflict of interest here?
The reason I raise this point is, this will be replicated again and again and again, as long as we bring bills to the floor that talk about financial services modernization and refuse to deal with the issue of thoughtful and sensible regulation of things such as hedge funds and derivatives and as long as we bring bills to the floor that say we can connect and couple, we can actually hitch up, inherently risky enterprises with the core banking issues in this country.
I hear about fire walls and affiliates, all these issues. I probably know less about them than some others;
Yes, I do realize it (Score:5, Informative)
Management of the leverage is what is missing here. NO FIRM should be allowed to leverage up 30:1. (30x your collateral)
You are comparing apples and oranges.
And since you went there with LTCM [wikipedia.org]....allow me to quote you a figure, "At the beginning of 1998, the firm had equity of $4.72 billion and had borrowed over $124.5 billion with assets of around $129 billion. It had off-balance sheet derivative positions with a notional value of approximately $1.25 trillion"
Management of leverage was obviously missing there. 1.25 trillion? That is absolutely totally fucking insane. But because Merriweather was a "smart guy" --- the investment banks let him do this. And it put the entire system at risk.
That's not supposed to happen. Ever. Yet it did. Do we learn nothing from history?
Re:The NYSE runs linux (Score:5, Informative)
That's nice, but I think he's more interested in analysis and management tools rather than actually running a stock market...
I think you're right. Here's a list of apps from something I just googled. [linuxlinks.com]
Re:Hmmm (Score:3, Informative)
Maybe, you need a little more knowledge of Economics in general and this particular sorry situation in particular, uhm? The failure of this company would've left millions of people uninsured, and hundreds of thousands unemployed. Taking it over, breaking it up, and selling the pieces at normal, rather than bankruptcy (a.k.a. firesale) prices seems like a reasonable thing to do.
I just wish, they would've done better to Lehman Brothers — instead they completely botched trying to sell it to Barkley's and BoFA. Instead of asking the would-be suitors to come up with bids in secrecy, the Feds allowed reps of both to be present in the room at once. Both balked... As a result, Barkley's is now taking over the juiciest pieces of Lehman's assets at a firesale price without assuming any of the liabilities — because Lehman is bankrupt as of Monday. Had they not known (from sitting at the same meeting, that Fed organized), that BoFA is not interested, they might've bet a fair price last Sunday...
Re:Let me save you the trouble (Score:2, Informative)
quantlib (Score:1, Informative)
not really (Score:5, Informative)
It's a complicated public/private structure, and basically anything less than a book-length explanation of it oversimplifies in one way or another. It's the de facto U.S. central bank, and the main federal agency regulating banking, but also has significant private components. Its funding sources are technically ownership and fees by various private banks, but restricted in such ways that it's de facto more like government regulation and bank taxes than ownership and fees between private entities. Its board of governors are appointed by the President, and confirmed by the Senate. It's also effectively an operational arm of the U.S. Treasury for everything from tax collection to paper-money supply to treasury-bond issuance and interest payments.
More to the point, its solvency is implicitly guaranteed by the U.S. government, much like that of Fannie Mae and Freddie Mac were (and those two quasi-private entities were actually de facto considerably more private than the Fed is).
Re:The NYSE runs linux (Score:3, Informative)
That comment really makes me wonder, what happens to the person who loans out tools to the gold miners for free ?
Lol, are you really so out of it that you think *that* is how it works?
If so, that would probably explain your lack of success in the business.
The majority of software development money is spent on non-cots/non-shrinkwrap software.
Nobody loans software, ESPECIALLY to wall-street.
Whether you write it on contract as Free or as proprietary, the important part is to write it on contract so that you get paid for the work you put in.
Re:Well, since I develop trading systems on FOSS (Score:2, Informative)
Full disclosure - I am a founder of a startup [marketcetera.org] that develops an open source automated trading platform targeted at institutional investors.
As was mentioned in above postings, there are a series of open source tools available to bootstrap your trading system development:
And then of course there's my company Marketcetera [marketcetera.org] - we build on top of a lot of the tools mentioned above and others (ActiveMQ, MySQL, Ruby on Rails, QFJ, etc) to provide the basic underlying platform that institutional traders (think quantitative hedge funds) can use to build their proprietary algorithms and start trading. After implementing a few trading systems in a row ourselves for various trading firms we realized that there was an obvious need for an open source trading platform so that people wouldn't have to reinvent the wheel and write systems from scratch every single time.
To answer the OP's question about which commercial firms use FOSS: - a lot of proprietary trading software is implemented on top of OSS - JPMorgan famously built their trading GUI [eclipse.org] [PDF] on top of Eclipse, and Progress Apama is built on top of Eclipse RCP [wallstreetandtech.com] as well.
Not surprisingly, most trading applications are very Windows-heavy (although quite a few companies have Linux clusters, and some exchanges run on Linux as well). Most of the apps that your broker will provide for you to trade with are Windows-only (such as Bloomberg, Goldman Redi, MicroHedge, etc), and a lot of the APIs available from vendors are .NET or COM components and nothing
else. We implement our systems mostly in Java (including the Eclipse RCP),
thought have connectors for some of the Windows-specific components.
We know that flexibility is at the heart of any powerful trading application, and we think the open-source model maximizes the ability of our users to control the application. Some think the open-source model is antithetical to the secretive finance industry, but we see it as the perfect fit.