Server Push For Applets?

John asks: "I'm designing a system where 10,000 applet based clients can view prices for a bunch of products. A requirement is that when a price changes, the clients see the new price within a few seconds. Basically, I'd like to push content to them. But a lot of them are behind firewalls, so a socket-based solution won't always work. OTOH, if they poll the server every couple of seconds using HTTP, that's 5,000 requests per second, which will require a server farm and multiple T1 lines. Does anyone know a more elegant way? I considered using HTTP requests that block in the server until a response is ready, but then I need to support 10,000 blocked threads and concurrent connections - another server farm. There's got to be a better way..."

tunneling RMI over HTTP

[The Mayor]

Let's see. As I see it, there are a couple of requirements:

-real-time update of events (or near real-time)
-client access from behind firewall
-minimize load on servers.

Hmm. This is not an easy thing to do. If you don't want to poll, you're asking for a stateful connection to the server. Stateful connections are *expensive*. They should be avoided if possible. How often do you need updates? Is every 5 seconds excessive? If you increase that to a minute, 10000 requests per minute is not too big a load.

Assuming you want to stick with a stateful connection, have you looked at tunneling through the firewall over HTTP? If you're doing this from an applet, I'm assuming you're using Java. RMI supports tunneling through firewalls using HTTP.

Why not use one of the application servers that supports JMS? JMS is made for this type of data connection. Much of the burden of optimization is left to the application servers, which is good (I don't have to worry about it) and bad (you only get the optimizations the app server vendor provides).

In general, though, for web applications, you want to use stateless connections whenever possible. This implies that polling is the preferred method. At any rate, any two of the above three requirements can be easily met, but the three together are contradictory.

-dan

Re:Resources required

[dweezil]

It's gotta be custom since you're going to be handling 10k idle threads at any one time.

A thread per connection model is probably not the best architecture for a system support so many users. See Dan Kegel's C10K site [kegel.com] for more info on high performance servers supporting many concurrent connections.

Server push

[Alex Belits]

Netscape implements "the original" server push using MIME multipart (see http://www1.netscape.com/as sist/net_sites/pushpull.html [netscape.com], section about server push). Even though no other client understands it, plugins or applets can just use the same protocol for this purpose, and it's possible to make server use internal support for it when the client understands it.

I use this in my webcam [denver.co.us] and X-10 devices control form -- compare form without server push [denver.co.us] (you have to press "Send" to get the updated status) and the same form with server push [denver.co.us] -- if you use Netscape (or recent builds of Mozilla), every change in the state of devices causes form to be updated at the time when it happened, and if some command is sent, first response represents the status after command is received (but not executed yet -- X-10 is slow), then after some time (can be half a second for simple on/off or few seconds for brightness change) when command is sent and confirmed, status updates using the server push. If someone else (or the same user from another browser, or me using "physical" control panel) sends some commands, status updates by itself without any interaction with the user or requests sent by a browser.

Re:Resources required

[Alex Belits]

You probably can't delay sending data by more then a minute or so, some firewall proxies may time out.

Firewalls can't time out in the middle of the response -- if the header (and possible some segments of multipart response) is already sent, proxy can't just drop everything. NAT however can, and this is why either server should send something small as a "hearbeat", or client should try to reconnect if server appeared to drop it.

I don't believe you need to do multipart/mime for this, since the proxies should _NOT_ know about the internals of HTTP beyond how to get it through the firewall. You're probably fine returning XML snippets for each update 'datagram' you want to send the client.

Proxy will just know that it is in the middle of some response -- it can't parse the internals when it doesn't know the type of data used, so unless it's badly misconfigured, it will work. XML can be used for data inside the segments, however HTML (for clients that support server push directly) or comma-separated lists (for custom applets/plugins) will be as good, and will take less bandwidth. OF course, inside segments gzip compression can be used, just like in any other response.

Resources required

[Harik]

Well, you're right on not wanting to force a refresh every second. Your best bet is a custom server speaking HTTP (for firewalled proxies) that sends a slow update back to the client. It's gotta be custom since you're going to be handling 10k idle threads at any one time. You probably can't delay sending data by more then a minute or so, some firewall proxies may time out. (Anyone with hard experience on this field, feel free to speak up)

The idea is the client sends a request, and waits for for the update. Probably initially it will recieve a dump of current data and display that, waiting for updates as they arrive. Alternately, you can send a sequence number with each update and the client sends it back. Any data that's updated after that date gets sent, again, blocking till the socket gets closed and sending as needed.

I don't believe you need to do multipart/mime for this, since the proxies should _NOT_ know about the internals of HTTP beyond how to get it through the firewall. You're probably fine returning XML snippets for each update 'datagram' you want to send the client.

IRC is a good model. 99% of the time, the client is idle, waiting for more input. It's a persistant socket connection with some header setup (User/nick/MOTD for IRC, URL/Metadata/HTTP headers for you)

Writing a custom server to handle this should be fairly straightforward. Watch out for those buffer overflows, though. ;-)

--Dan

HTTP pull is fine!

[hatless]

What's this nonsense about server farms and giant servers for 5,000 requests per second?

Your network traffic and the burden on the networking resources of your servers is the same regardless of whether you do push (ick) or pull. The key is in how hard you make the servers work to handle the requests.

What you want to do is cache the content and use dumb servers to push all the data out to clients. Something architected like the $100,000 Vignette Storyserver or the version of the free PHP FastTemplate package with caching added are the right idea, though you could do it with any reasonably lean server technology (in other words, CGI would be bad.

Presumably, you have a finite set of data snippets, many of which go to multiple users. So generate it once, store it on a filesystem, and leave it to one dumb, unthinking HTTP server and one or more proxy caches (like Squid or a commercial service like Akamai) in front of it to hand the static files out. Do your refresh, and so forth. If there's been no change in the data, don't update a given snippet file, and the servers' cache management and the web browser will only do a HEAD request and move on.

If you want to cut network traffic by adding bigger hardware, you could make your client request multiple snippets in a single request and use something as simple as server-side includes to concatenate the cached content snippets into a single returned file.

This sounds like a project that's generating revenue, either directly or indirectly. That should result in a budget for this.

going to keep this vague...

[kootch]

but how about a verification # on a content library that will determine whether the current iteration of the data matches the data that currently resides with the user.

upon determining whether the # matches or not, the information that was changed will be marked as changed, and only the fields that were marked will be sent to the client.

more processing, smarter system, but less bandwidth?

Why push?

[demin]

First of all, you will not win much with push, unless you use some sort of multicast. I guess, this is hardly possible, but you can think of it. Second, you can utilize async I/O web server. With web server like Zeus [zeus.com] you will have 10000 sockets rather than 10000 threads. I didn't check it out, but this supposedly can be done with a single box. Third, since you are dealing with applet, you can use CORBA deferred synchronous request/response, wich possibly does exactly what you want.

Threads and concurrent connections

[Alex Belits]

There is no way to avoid keeping idle connections open if the data can arrive at any moment. However with proper design (that means, not Apache -- callback-based model won't work well there) you don't have to waste any resources on them except a file descriptor and small amount of memory for some internal structures. Having large number of mostly-idle tcp connections is not something unheard of -- any large IRC server does that for thousands of idle users -- however the design of most HTTP servers is not optimized for this.

One of the goals of my fhttpd [fhttpd.org] server design was to separate the model of request processing in the backend from internals of HTTP server itself, and the examples in my previous messages are running on custom backends written for fhttpd -- webcams are always six processes without any additional threads (three processes per camera -- one process talks to http server, one talks to the camera, one compresses jpegs -- the program is released as qcwebcam [fhttpd.org]), and X-10 is handled by a single process (also with no threads involved) that handles everything and talks to X-10 "modem" through a serial port.