How to Work Around Broken Port-80 Routing? 326
Dr. Zowie continues: "I use a regional ISP with otherwise-very-good policies. However, they seem to be intercepting
anything that comes from my home net on port 80, so that they can
``transparently'' cache web requests based on the payload of those
packets. The proxy seems to work rather well in most cases: I
never noticed it until I started using OpenNIC. Then I found that some web pages that should have
resolved OK through the OpenNIC system failed even though routing on
different ports worked OK.
"I did some experimentation using ``telnet'' on port 80
directly, and found that packets are being routed based only on
the payload regardless of the original destination address: I can (for
example) retrieve the Slashdot front page by using ``telnet
www.google.com 80'' and asking for "http://www.slashdot.org
http/1.1". The tech support folks seem to be stonewalling me: the
main contact tells me that the behavior is "not broken" even though it
clearly violates RFC
1812, the standard set of rules for IP routing.
"The practice of ``transparent'' proxy routing seems to be growing
more widespread. It appears to break the internet standard in a way
that works for most folks for now, but that breaks port 80 usage in general. Looking ahead, this breakage seems
like a growing nightmare waiting to happen. At the very least, I
expect more instances of my particular problem to appear as folks give up on the corporate hegemony of ICANN. More insidiously, transparent
proxy routers break the layered nature of the internet protocol and
restrict the flexibility that made it work in the first place. One would
hope that such proxies would at least act like routers when the fancier
proxying fails, but at least my ISP's doesn't. What about your ISP's?"
Use netcat... (Score:4, Informative)
Re:Use netcat... or your own proxy server... (Score:2, Informative)
Re:Use netcat... or your own proxy server... (Score:3, Insightful)
How could a number outside 16 bits make it to a router since TCP only holds 16 bits for ports? If you wrap around to 80, you have 80, not 65616.
-Kevin
Re:Use netcat... or your own proxy server... (Score:2, Insightful)
Re:Use netcat... or your own proxy server... (Score:2, Informative)
Re:Use netcat... or someone else's proxy server (Score:4, Informative)
You could also use a third party proxy server. You can find gobs of them here:
http://tools.rosinstrument.com/proxy/
and here:
http://directory.google.com/Top/Computers/Inter
Tunneling (Score:3, Interesting)
one possible solution (Score:2)
then use that machine as a web proxy, or set up an ipsec tunnel to that machine and route your port 80 traffic through that tunnel.
Sounds like what my college does (Score:3, Informative)
[OT] removing Konqueror from KDE (Score:2)
Not the removal, the separate availability. <mind mode=screensaver>You should be able to buy Linux and install it without Konqueror, and Konqueror without Linux. Oh, wait a minute... you can!</mind>
Just to rub the point home, you can buy and install Linux with or without graphics, with a different Graphics layer (such as Berlin [berlin-consortium.org]), with a different window manager (such as FluxBox [sourceforge.net]) and so on. All (modulo a few libraries) with or without Konqueror or one of a host of other browsers (Mozilla/Galeon/SkipStone, Netscape, Opera, Amaya, Mnemonic, OmniWeb etc).
same problem (Score:3, Interesting)
Don't ask me why
Re:same problem (Score:2)
Education (Score:3, Interesting)
my friend set up an anonymous web surfing proxy at his home computer, and using this i can get whatever i want.
there are publically available anonymous port-80 proxies still around.
Re:Education (Score:2, Insightful)
1. Refuse to use the machines at school for any internet access. Period.
2. Let the board and the teachers know why. Tell them they've taken a good thing and turned it into a complete waste of tax money by senselessly restricting.
3. Ask the board why they think their current system is capable of making better judgements than their salaried teachers.
This is probably why I really didn't get along with anyone in high school. But this stuff really ticks me off - usually some overzealous admin taking the liberty of forcing his/her idea of "good" on to everyone.
Re:Education (Score:2, Interesting)
The school board of any decently large school district is generally disinterested in actually educating students or making things work well, most school board members are just there as a springboard for higher political office. They generally don't give a fuck about the students or education, so they lower test standards and claim that test scores have improved. If it makes them look good to "protect our children from evil" by blocking out these sites, they'll do it. School systems don't operate like normal organizations; the students' opinion carries next to no weight at all, as "it is up to us adults who know better to protect the students from what they don't know." Total BS, but that's the way public school works.
My Experience on ISP with faulty service (Score:5, Interesting)
The thing is, they probably won't listen to problems like this, or your proxy issue in most cases. But I found a way to make them listen to you:
Phone them up saying that you want to cancel the service. Mention something about their web hosting being broken. They will probably say that they will have a management person phone you back to confirm the process.
When they do phone back, for me, the call was like "Hello, there was a call eariler about a slow connection?" And at this point you have someone on the line who is interested in helping you, has power in the organisation to really fix things (because they're management or a senior tech) and they want to get your issue fixed to they don't lose your business. And THIS is when you really try to explain what's going on.
This was my experience. Perhaps it will work for you.
Re:My Experience on ISP with faulty service (Score:2)
This is a method of getting things solved that really does work ... but generally only with smaller businesses. Before I went to work for an ISP, I had an account with another ISP that was doing an excellent job. About a month into the job I called up the other ISP to cancel my service (as it was a dedicated full time nail-up service, and thus a bit more costly that plain old dial-up). The lady that answered took my request and proceeded to ask why I was dissatisfied with the service. I then said "Oh, no, I was perfectly happy with the service". She then asked "Oh, you're moving out of town?". To that I finally explained to her that I had taken a job with an ISP in town, and had free service through it. She told me "well, I guess it won't do any good then for the owner to give you a call and see if he can get you back".
You're not likely to get that kind of response from a big national ISP. One customer is just too small a percentage for a manager to call to find out why you are unhappy. Mostly, whatever changes they might need to do to make you happy would be too costly for such a big company, anyway. The trouble with small ISPs is that there is such a great variation of competency. But when you get a good one, you have a gem. With the big ones, it's generally a fairly uniform level of pathetic service.
Lots of ways to work around your ISPs. (Score:5, Informative)
Bouncers - You set this program on an external server on a port thats not filtered. You just point your browser at this IP/port and your outside your filtered isp. Check www.freshmeat.net [freshmeat.net]
SSH, tunnel or route from an external box.
Really, If you cant go through it, go around it, either with software or networking.
-
Well, if crime fighters fight crime and fire fighters fight fire, what do freedom fighters fight? They never mention that part to us, do they? - George Carlin
Proxies have similar problems - try tunnels (Score:2)
If you can use a tunnel server, like IPSEC or PPTP or SSH, which lets you pick the IP address to send your IP packets to but doesn't interpret the packets itself, you'll mostly be ok (you'll still have to make sure to do your own DNS if you want to resolve on alternate roots.)
Their way or the highway (Score:3, Informative)
(2) Talk to your ISP's sales department. Tell them your problem. Tell them you're ready to move. (Perhaps ask what the hit rate of the cache is, that is, if the overhead is worth it for them.) See if they offer any accomodation.
(3) Go with the ISP that does what you want.
If you're using them for DSL, you may not have a lot of choice.
(As others suggested, if host resolution is your issue, you could run a local proxy on your 127.0.0.1 interface that converts host names into addresses.)
Re:Their way or the highway (Score:4, Informative)
Unfortunately, that's not a complete solution. Example: Compare my home page [jerf.org] versus the IP address [65.196.231.181] that hostname resolves to.
Lots of servers do this.
Re:Their way or the highway (Score:2)
Re:Their way or the highway (Score:2)
As far as a local proxy: that won't work with virtual hosting in a non-ICANN name space. The immediate problem is that I can't retrieve non-ICANN web pages because the proxy tries to resolve the non-ICANN name in the payload, using ICANN DNS. I can always ask for numeric addresses, but virtual hosting (where a server gives you different pages depending on the name you ask for) is widespread enough that there are many web pages I can't retrieve even in principle.
Cheers,
Craig
Look At It From the ISP's Standpoint (Score:5, Insightful)
99.9% of the ISPs clients aren't trying to do anything tricky, like this. Of those 99.9%, say, only 40% have a proxy server specified. These 40% get to enjoy faster web browsing--which is probably all they're doing anyway. The other 60% enjoy slightly less quick web browsing, but that's they're own fault, right? They're the only ones losing out, right?
Wrong. The ISP has to pay for bandwidth. The ISP doesn't like the proxy only because it makes browsing snappier, it likes the proxy because it also saves them on bandwidth costs! If the other 60% of the clients were using the proxy they might save 10%, or more, on total bandwidth costs.
You could think of it like this, too: that's 10% more bandwidth available for the clients at no additional cost to the company (apart from the capital for the proxy server). Yes, they're not perfect, but they make a difference. When you weigh the pros and cons, well, it's obviously going to be worth it for the ISPs to have it installed.
You could look around for an ISP that doesn't use a transparent proxy but, as you said, they're becoming more popular. Realise that they're not doing to squash your freedom, but instead to provide better service and to save money.
Re:Look At It From the ISP's Standpoint (Score:2)
What I'm complaining about is that the proxy has a bad failure mode: when the usual resolution method fails, the proxy ought at the very least to fail softly and route the packets like a real router should Unfortunately, it doesn't -- it generates an error message instead.
Re:Look At It From the ISP's Standpoint (Score:4, Insightful)
2. What is happening is that your *default gateway* (which really IS a router) is redirecting packets bound to port 80 to the proxy server. Your default gateway is doing the routing, NOT the proxy server. (Linux does a nice job at transparent proxying, btw.)
3. The proxy server then tries to resolve the domain name using DNS.
4. The DNS server the proxy server is configured to use, not knowing anything about these funky TLDs you're trying to access, can't find it. It tells the proxy server so.
5. The proxy server comes back and gives you a nice, friendly error message telling you it can't resolve the host name.
Look...transparent proxying is to bandwidth what NAT is to private networks. It works, it works very well, it's in widespread use (getting wider every day, probably), and it's here to stay. If you really want to do something constructive to solve your problem, ask your ISP to configure their DNS to resolve the OpenNIC TLDs. They're a lot more likely to do that than they are to stop using transparent proxying (I know I would be).
Re:Look At It From the ISP's Standpoint (Score:2)
That's not necessary. In the case of a transparent proxy, which has intercepted the IP packet, the IP address of the origin server is available. It just has to be coded/configured to use it properly. In the case of the server running at peakpeak.com, this is not the case.
Also, this will not poison cache. You cache indexed by a tuple combining the hostname (if provided) and the IP address. The cache of several IPs can be combined in the case of DNS providing multiple IPs for the hostname. Thus if example.com resolves to 1.1.1.1 and 2.2.2.2 and 3.3.3.3 then all cache lookups that ask for example.com+1.1.1.1 or example.com+2.2.2.2 or example.com+3.3.3.3 can share the same cache subtree. But if some client asks for example.com in the HTTP "Host" header while connecting to 4.4.4.4, the cache lookup would be for example.com+4.4.4.4 which would be a separate cache subtree from the other 3.
Re:Look At It From the ISP's Standpoint (Score:2)
Even without WCCP, doing a straight route through a cache server box preserves the destination IP address in the IP header ... it has to or packets cannot be delivered. You can also do a route-map based on destination port number and route only TCP packets destined for port 80 to one box and default everything else as usual. The destination IP is still preserved (but the ethernet MAC address is different as the routed-to IP address is used by ARP to find that MAC address). Now whether Squid can handle this case or not I haven't tested. But I do believe WCCP encapsulates the original destination IP address, so there's no reason not to correctly associate it with the request, and correctly connect to the intended origin server, and correctly distinquish cache data by IP address (which can be shared among those that DNS says are a common group with multiple A-records).
Re:Look At It From the ISP's Standpoint (Score:3, Insightful)
I agree with everything you say; proxy servers are a great thing for all involved and not a threat to freedom.
But the problem is that this proxy server doesn't work right. My browser should look up the IP corresponding to the site, send a request on port 80, and get the response. In this case, it looks like the proxy is insisting on doing the lookup part, and so the user effectively can't change his DNS.
Re:Look At It From the ISP's Standpoint (Score:2)
Or the OP could look around for an ISP that does use transparent proxying correctly. This is not an unsolvable problem; all the proxy has to do is connect to the correct origin server, which is the IP address the client connected to. This is necessary because with transparent proxying, the GET request provides a URI, not a full URL.
Re:Look At It From the ISP's Standpoint (Score:3, Insightful)
Well, in that case, they can stop supporting anything but windows, since it has a clear majority. Oh, and you can't use anything but IE since it's got a majority as well.
The problem is that I don't pay for 'a service that allows me to view most web sites'. Rather, I pay for an 'internet service'. If anything that should work, doesn't, then they are violating their end of the contract... Not to mention probable false-advertising, etc.
If it costs them 10% more bandwidth for those who choose not to use their optional proxy, then they should charge the customers 10% more.
How about if the USPS decided to crush every package by 1cm because then they can fit more packages in each plane/truck. Besides, 99% of people have at least 1cm of padding to protect the package contents anyhow.
It's exactly the same thing. Doing something that doesn't hurt too many people, in exchange for more profit. The fact that most people aren't going to be negatively affected doesn't make it right, or legal for that matter.
Re:Look At It From the ISP's Standpoint (Score:2, Informative)
ISPs required by law to block port 80 in Singapore (Score:4, Interesting)
This has caused me many problems before, when my IP gets determined wrongly by the remote site (which naturally thinks takes the proxy server's IP for my IP address). Some applications don't like the transparent proxy either, for example Frontpage Extension (not my choice to use!), and an autopatching program which refused to download the latest version of a file, insisting on downloading only the file cached in the proxy server until the cache gets flushed.
The only real method of bypassing the proxy is to use another proxy server (since 8080 isn't blocked) outside the ISP's network. This tends to be really slow though.
I guess I have to live with this until the government one day realises that proxy servers cannot stop the people from viewing pr0n, and it's probably not worth maintaining the proxy servers to meet the demands of all the net users in Singapore, not to mention maintaining the list of sites to block.
Re:ISPs required by law to block port 80 in Singap (Score:2)
OK, this is a bit OT, but since you're from Singapore, I'm curious about something. I know that when filtering was proposed there, many people weren't happy about it. Has there ever been a move to form something akin to the EFF to protest this, or is the political situation still such that doing this would get you hauled into court by the government?
The whole political situation there baffles me. More repressive governments have been forced to reform by popular protests. Why hasn't it happened in Singapore? You'd think that, with the extent to which the country is connected to the rest of the world, people would see what's happened in places like Indonesia, Thailand, Yugoslavia, etc. and want to do the same.
Re:ISPs required by law to block port 80 in Singap (Score:2)
Re:ISPs required by law to block port 80 in Singap (Score:2)
The Singapore government is probably more concerned about stopping people accessing the numerous overseas sites run by the opposition movement. For those that don't follow Singapore politics it is one of those countries where the government brings specious lawsuits against opposition politians and elections are run in the manner of the old Soviet Union.
Of course since it is a capitalist pseudo-democracy this rarely gets comment in the western media. When it does the government has sued for libel under its mickey mouse libel laws in its kangeroo court system.
All phone calls made in Singapore are tapped and the government analyses the telephone call logs to see who is talking to whom. Its kinda the state that Ashcroft would like.
corrections, suggestions, etc (Score:5, Informative)
Second, there's a lot of ways around it which involve tunnelling.
Tunnel to another box running a non-broken web cache. I used to tunnel my http traffic through ssh to my colocated boxes, which ran adzapper, and proxied through that.
Tunnel at the IP layer by running any IP-in-IP encapsulation. If you have some version of windows, for example, you might convince someone with a server to run a PPTP server for you somewhere and you could tunnel through that. There are even Free PPTP Servers for Linux [poptop.org] available to help.
Find someone who runs a little proxier for their own net with socks, and bounce off their socks proxy. Someone you know no another ISP probably has Wingate or the like running, and if they allowed it (and on some older version, it will permit this by default), you could set your browsers SOCKS settings to bounce off their proxy server, and since SOCKS isn't on port 80, your ISP will probably ignore it.
There are also a number of things you might discuss with your ISP to resolve the issue.
Suggest that they switch to a less broken cache server. (Squid [squid-cache.org], anyone?)
Suggest that they exempt you specifically from the cache server by telling it to ignore your ip address.
Note that they have an obligation to make sure their caching software doesn't interfere with your browsing; so it will be necessary (and not cost-effective for them) for you to call for every problem you notice.
Obviously, you'll need to probably speak to a whole number of supervisors, and probably eventually get transferred to a "real engineer", and they will probably hack in a fix (like exempting you only) rather than truly deal with the problem.
If all else fails, then you may want to try issuing ultimatums, like, "If you can't fix this problem, then you can cancel my service." Tech support people are lazy, however, in some cases, and may just opt to cancel you. This is a harsh reality in the world of consumer bandwidth -- and it will be worse, soon, with bells closing their DSL lines to competition, meaning unless someone else builds a telephony infrastructure to you, you'll probably pick Cable vs 1 DSL provider, and if you don't like something at either of them, you're just out of luck.
Re:corrections, suggestions, etc (Score:2)
>misnomer. Web caching is something that happens
>on the application layer of the OSI model,
>layer 7, whereas "routing" refers to layer 3,
>which supplies IP routing for the TCP/IP
>protocol suite. What's broken is their caching,
>their cache server, or their proxying; pick a
>term.
Thanks for the helpful comment!
What I'm complaining about is that their router (layer 3) routes all port-80 packets to a cache server that looks at the payload only (layer 7) and not at the header at all. In short, they're not routing correctly; they've broken the layered structure of the protocol.
Re:corrections, suggestions, etc (Score:2)
Be sure to try the other helpful suggestion I read of trying port 65616 (that is the right port, btw) -- if your proxy server is stupid, it might pass that on. Of course, you'd have to type it into your URLs a lot, but it is still a way to get around the cache when you need to, if it works.
Re:corrections, suggestions, etc (Score:5, Interesting)
Au contraire. Tech support people are tired of listening to customers whine about problems that tech support people cannot fix. If customers have unreasonable expectations, and refuse to listen to us, it's far better for the company if they just cancel service and go elsewhere (becoming somebody else's problem).
Also, non-chalance about canceling service is sometimes the best way to make customers understand that we really are doing our best to help them, and we're not just blowing them off. Sounds weird, but here's an example:
Customer has a problem with their DSL service. We've identified that the problem lies with the phone company. Phone company has given us a commit date of Tuesday by end of business day for repair to be complete. For whatever reason, the customer feels like they've been dragged around, and their service isn't getting fixed. Customer says if they're not up and running by 9:00am Monday morning, they're cancelling service.
Customer expects us to bend over backwards to get them up and running by 9:00am Monday morning. We can't. There is absolutely nothing we can do. It's out of our hands. Customer needs to understand this. Customer will have the same problem at any competing DSL ISP, but we're the ones who have identified the problem and are getting it fixed.
We respond by repeating to the customer that we have been given a commit time of Tuesday by end of business day, but that we cannot guarantee that the issue will be resolved by then. We then offer to the customer that if this is unacceptible and they'd prefer to cancel service, although we'd hate to lose them as a customer, we'd be more than happy to transfer them to someone who can take care of that.
This has the effect of making it clear to the customer that we really mean what we say. Usually, they shut up, keep their account, and let us do our jobs. Often, they'll ask to be transferred to get the account cancelled, then hang up during the transfer.
The alternative is to offer the customer incentives to try to convince them to stay with us, such as offering a free month of service, or a credit on their account. This costs us money, and gains nothing - if the customer has the expectation that we're willing to give him free service, he'll try to take advantage of it in the future. Far too many ISPs have failed for this very reason.
At the last few ISPs I've worked for, nearly all my coworkers have been genuinely interested in helping the customer, and we've been fortunate to have management that allows us to do so. I understand that at some companies this is not the case; those are obviously the ones to avoid.
Sorry for ranting. Getting back on track: ultimatums like "if you don't fix this problem, I'll cancel my service" sometimes are a good idea. That will tell you whether or not you can get the issue resolved. Be prepared to actually cancel, because if they can't resolve the issue, that's what will happen. If they can but just don't want to, threatening to cancel may just be the incentive they need to get it done.
Re:corrections, suggestions, etc (Score:2)
I could relate anecdotes all day, but here's a good one: customer calls up, and is getting the message, "Out of memory" and the app is crashing. Customer only has 4 or 8 megs of ram, a smidge low but not too low to run the app back in the world of WFG 3.11. I happened to divine this from listening to the call from the next cube over, and heard the person conclude that windows was causing the problem and recommend to the customer that they reinstall! ARGH. I practically jumped over the cube wall and got them to transfer them over to me, where I discovered that the customer had their virtual memory settings set to 0MB, manually. That and the 4/8M of physical ram did not mix.
"Reinstall Windows" and "Reinstall XXXXX" (our product) were the favorite responses, and it was a culture of lazy, voodoo support, which is why I got customers calling back for the 12th time, sometimes. Add that to a hold time of over an hour, and its no wonder that company faded into obscurity.
My experience going the other direction is even more common. I can scarcely EVER call anyone responsible for supporting an Internet service and not get either (1) a moron or (2) a lazy moron. Admittedly, I don't expect a lot -- good engineers don't like doing technical support. But when you describe routing problems to your colo provider and their support staff tries to immediately claim it is a problem with some other provider before even checking a traceroute (which, if they knew what they were doing, would let them realize what an out-and-out fabrication their claim obviously was), you get fed up.
However, all this aside, my advice to the original poster was based on this general idea: his support people will be familiar with stuff like checking DSL line-downs and scheduling truck rolls, debugging customer DNS/DHCP settings, etc. They will never have any reason to even wonder about a caching server, most likely, but because many tech support departments have poor to nonexistant escalation paths to engineering, they will attempt to shaft the customer, either knowingly or perhaps ignorantly. But one way or the other, they won't fight their way through to escalate properly. I think if your experience is different, its the exception rather than the rule. (and bravo to your TS department if you do provide competent service time and again -- I've yet to come personally in contact with anyone who could do it, including a company where reps made an average of over $55k/yr).
Re:corrections, suggestions, etc (Score:2, Insightful)
In my example, the source of the problem was the phone company, not the ISP. In the case of most large ISPs, the ISP has a contract with the phone company wherein the ISP orders the DSL service from the phone company, and the end user doesn't talk to the phone company. In this situation, an end user cannot even talk to the phone company about DSL problems. You can complain to your local Public Utilities Commission, and I would highly encourage you to do so.
Should a customer just grab their ankles and say "please sir, may I have another".
In the United States of America, with Republicans running the government? Yes. Congress is currently trying to pass legislation to make the situation even worse.
In other situations, where the problem does lie at the ISP, the best place to go is straight to the top. Try to get in touch with the president of the company, or somebody else in upper management. They have the power to make it happen. Do not try to work your way up the chain (don't try to go any higher than the direct supervisor of the tech who answered the phone; that will get you nowhere); you have to go to the executive level.
Poor ISP. They get money to do a job and they can't. So some pissant customer support moron complains about the customer.
In my example, no competing ISP can do the job any better, except possibly the ISP that is owned by the phone company, and they do the rest of the job so poorly it's not worth it. In the case of problems at the ISP, understand that the company's management may not even be aware that there is a problem, and tech support often can't fix it. Complain to management, let them know there's a problem. If that doesn't resolve the issue, take your money elsewhere. That's capitalism at work.
Poor you. You have a job that you can't do, so its the customer's fault. Boo hoo hoo. I see a long career at McDonalds for a slacker like you.
I never said anything was the customer's fault, nor did I say I can't do my job. Sometimes telling customers "no" is my job; I'm not slacking when I do it, and it's not the customer's fault I have to.
The REAL solution (Score:2)
The BEST solution that unfortunately will never be implemented is to allow specifying a port number in a DNS lookup. Then when the browser or e-mail looks up the address, one could also specify a port that you want.
Unfortunately, this ain't gonna happen without a rewrite of everything.
Re:The REAL solution (Score:2)
Actually, I mis-read his question. Apparently his problem is outgoing port 80, rather than incoming port 80 block, which is what I'm addressing.
What brand of transparent proxy is it? (Score:3, Interesting)
Of course, the problem with transparent servers is when they're not, and your ISP seems to have one that isn't. Is it possible to find out what kind it is, either by telnetting to the thing and looking at headers or by asking the ISP, and can you do bug reports to the vendor to get them to fix their product?
Some issues can be worked around. (Score:2)
Re:What brand of transparent proxy is it? (Score:2)
Why do you think it cannot be worked around? Do you know what brand this proxy is? Do you work for peakpeak.com? Do you work for the vendor of that proxy? It sounds to me like someone set up a non-transparent proxy in a transparent role. But I would not rule out some vendor marketing department over-hyping some product as a solution for transparency when in fact it is good only for caching.
This saves LOTS of bandwidth (Score:4, Insightful)
There have been problems with the proxy in the past (it not returning any data) and there are still some minor issues, but on the whole it works well (in that you don't ever notice it).
It sounds like the ISP in question has a bug in their web cache code. If the web cache doesn't have the particular URL cached, it forward the request to the intended destination. I'd bet it's trying, but it can't lookup whatever OpenNIC URL is being specified (because it doesn't use OpenNIC). The ISP really should report this bug to the manufacturer.
My advice is this -- get the ISP on your side to fix the problem. They won't remove the proxy, and they shouldn't have to if the bug is fixed.
Re:This saves LOTS of bandwidth (Score:2)
Proxies are not inherintly bad. As you point out, they do have important benefits. But a broken proxy, as peakpeak.com has, which fails to make the connection to the IP address the client tried to connect to, causes problems. The correct behaviour would be to do the DNS lookup to get a valid list of IP addresses. If the intended origin server the client tried to connect to is among those, then check the cache by hostname alone. If the IP addresses do not match, then check the cache for a combination of hostname and IP address. If the cache doesn't have the requested document, then connect to the proper IP address and fetch it.
I don't know which proxy servers (can) do this properly. Hopefully someone can post (a link to) a proxy server review with this much detail. I do agree the ISP needs to fix the proxy, and it shouldn't be necessary to remove the proxy to accomplish it. If they can't fix the proxy, then a workaround is to route the one customer around it (route-map in Cisco IOS can do this).
Well, some bandwidth but it works well (Score:2)
I think the byte savings isn't quite as good as that, but I don't have any solid data to back that up.
The best I can say is that we had to shut the cache off for a day or so to do some maintenance and the help desk got a lot of calls about how "slow" the web was, in spite of the fact that not more than a few days prior we had *doubled* our internet bandwidth (single 1.5Mbit frame to MPP bonded dual pointtopoint).
I think that overall it provides much better bandwidth utilization (ie, fewer packets on the ISP link, even if the byte savings is only 10-20%) and the client browsing experience is a lot snappier.
Our ISP used to have a whole statewide squid cache hierarchy which you could tune your local squid cache into if you wanted to -- I wish they still did, the aggregated caching would have been very nice.
AOL ignores ports (Score:4, Interesting)
I stumbled across this when their proxy had some trouble with the cookies we were using and suddenly no one on AOL could use our service. A few minutes later they could again. Then they could not. During this time, I was running a packet logger on the outgoing traffic from our server and on the incoming traffic to a workstation I had connect to AOL. Everything worked find until the server sent the cookie. Then AOL suddenly stopped sending more packets. This occured on every port I tried, even ports reserved for other services.
The behavior is correct. (Score:5, Informative)
In other words, it's your client that's broken. See RFC 2616 for details.
The unfortunate truth is that more often than not, sites simply don't set their cache controls correctly. They forget that caches don't exist just on the server side but that they exist on the client side as well. Section 13 of RFC 2616 explains how they work in great detail and it really should be mandatory reading for any site administrator.
If you're still looking for more information on web caching, check out Content Delivery Networks by Scot Hull. It was just released and is available on Amazon. There is an enlightening section on web caching that should clearly explain why what you're seeing is in fact correct behavior.
Re:The behavior is correct. (Score:3, Insightful)
It will try to resolve the address in the GET line, and fail, because it doesn't know about other TLDs.
The only way to fix this broken proxy behavior is to have it ignore GET lines that is can't resolve, and instead forward the request intact to the IP address.
Re:The behavior is correct. (Score:4, Informative)
I'm not familiar enough with the ins and outs of cache design to know whether RFC2616 is designed primarily for ``transparent'' or ``selected'' proxies, but using a DNS resolution on the destination host seems to break the layered structure of the IP stack. In this case, packets that I've (layer 3) addressed to a specific host never get there, because (layer 4) they're being directed to another machine based on port, and the other machine (the cache) is routine them based on a name (layer 7) contained in the packet payload.
That is acceptable behavior for a proxy to which I'm explicitly routing my http requests, but not for a router down which I'm sending port-80 IP packets.
Wrong. That's not what's happening. (Score:2)
Wrong. That's not what's happening. Ordinary proxying does use the modified GET request form where the URL is used in place of the URI. However, transparent proxying is different because the client is sending a URI, not a URL. And it's connecting to the origin server IP address directly, not to the proxy. The only way to identify the correct host is to use the IP address the client attempted to connect to. That's the transparent in "transparent proxy".
If a client does attempt to connect to some IP address, and a transparent proxy won't use that IP address because it thinks the origin server is at another address, that's wrong. But if it has no idea what the origin server IP address is at all, even though the client was indeed connecting to it, then that's doubly wrong. A message from the transparent proxy saying it cannot find the IP address is simply stupid because it has the IP address the client connected to, since this is a transparent proxy.
How to find a transparent proxy's IP address (Score:2, Informative)
If you want to find the IP address of a transparent proxy, simply point your web browser at a web page that will print out "your" IP address when you request a web page. Instead of printing the IP of your firewall or your host, it will print the transparent proxy's IP address.
For example:
After that, you may be able to do some more investigation into what kind of host it is and/or what kind of software it is running. (This is left as an excercise for the crac...err, reader.)
Re:How to find a transparent proxy's IP address (Score:2, Insightful)
Re:How to find a transparent proxy's IP address (Score:2)
But will your transparent proxy connect to the exact same IP address that your client tries to connect to, regardless of what the "Host" header in the HTTP request identifies for the virtual host selection within that server?
It's in the layers. (Score:4, Informative)
What happens in your case might be that they have placed a switch that can do at least layer 4 switching, between you and the internet.
What then is done is that all port 80 requests coming from the clients side(you) are re-directed to the proxy which means that http requests on other ports will not be cached. Note that anonymous ftp can also be proxied.
A "clever" proxy/switch solution can do ip-spoofing so the webserver gets your IP adr. and sends it back to you directly, but as there is a switch inbetween, it redirects the result to the proxy which then sends the result back to you.
A way to avoid it is to get a gateway somewhere that can channel your http traffic, you could set your browser to use this gateway as a proxy on any port. The switch will most likely not act on the traffic coming on this port an pass it though.
The easy way would be installing a proxy server on a box that you have access to on the outside and configure it so that it won't cache anything.
How can you detect transparent proxying? (Score:2)
How can you detect transparent proxying? Or opaque proxying?
the best defense is a good offense (Score:2)
a non-broken proxy cache. Technical solutions are
nice, but they only fix the problem for *you*. If
you care about your peers, and the community of
users, solving the problem for *everyone* is much
to be preferred. Most users won't even understand
that they are being screwed by the ISP. They
depend on you to resolve the issue. Keep calling
support until they fix it.
Not 99.99999% (Score:2)
I doubt it's 99.99999%. There are apparently quite a lot of people trading around on the .MP3 and the .DVD hidden domain networks. RIAA and MPAA people most likely have no idea how to get there, if they even know it exists. Do you?
Comcast's Proxies (Score:2)
Bandwidth controls (Score:2)
My previous ISP, Sympatico [sympatico.ca], used to have a transparent interception caching proxy. It was quite troublesome and more of a translucent crashing poxy server. I remember being unable to access starwars.com for two weeks once, even though everything else seemed fine. It was particularly annoying for people whose MTU was set too high (they needed 1454 or less) as they would constantly get timeouts on HTTP POST, such as when trying to send email from a web interface like Hotmail or Yahoo. It was also a constant source of problems for people trying to author their own personal web pages as it would cache them and not show their updates.
My current ISP, IStop.com [istop.com], has an optional proxy. This is great! I normally use it, but if I have problems, I can switch to a direct connection. They run Squid and they also seem to have some sort of advert filter running. I get their logo (cached by my browser) or "This ad zapped" messages in place of at least 80% of web ads, which saves me lots of irritation, and both of us save lots of bandwidth. Incidentally, they also have reasonable bandwidth caps: 10GB non-local + 10GB local (mail, news, proxy, etc) per month, with excess charged at C$3/GB.
After a while, Sympatico reduced HTTP interception to large population centres like Toronto, Montreal and Ottawa. Finally, they stopped doing that too. I guess it was causing too many problems and costing them too much to deal with it. If my ISP were to introduce an interception proxy today, I would leave them immediately. It's just not worth the irritation and problems for the length of time it will take them to fix it or get rid of it. I do live in an area where there is plenty of DSL competition though.
So that would be my advice: switch ISPs immediately. Don't waste anymore time or effort on these guys.
Have you tried asking them? (Score:2, Interesting)
For more than 99% of our users, they don't know what routing or cacheing is, much less that it's happening. For those that actually have issues with the proxy it's a quick modification to our ipchains rules. So far we've only had 2 such requests. Also, we disable the cacheing for business class users by default.
I would hope that you would ask them to disable their transparent cacheing for you before doing something as rash as dropping them. It's my bet that most of their other users do not have this issue, and they may not even be aware that it is causing problems for you.
Err...so what is broken exactly? (Score:2, Flamebait)
From the submission, it actually appears that the proxy is working exactly as configured. The end user, however, is breaking things himself by using nameservers other than his ISP's. That can't be described as a failure of the ISP by any means.
Proxy servers add a lot of value to any network larger than, say your 3l33t home rig. The two main purposes I use them for are to reduce overall bandwidth usage, and to insert some level of malware protection. I've saved myself, and my company a lot of headaches by blocking silly virus code requests.
It's nice that the post managed to include links to RFC, etc... it's too bad that they don't seem to really have an understanding of how networks, specifically the Internet, works.
As others have commented there are plenty of alternative ways to get around this like SSH tunnels, VPNs, third-party proxies, etc...
Just my own little $0.02 worth of a rant. Please drive through.
-buffy
Re:Err...so what is broken exactly? (Score:2)
That's the problem with a lot of software "solutions": people don't think through all the worst case scenarios, just the main one, and the tool breaks when it's used in a way they didn't expect.
Re:Err...so what is broken exactly? (Score:2)
What I'm asking is to describe what you're seeing that doesn't work through the proxy. WHAT is broken? Are you having a problem connecting to a specific site, or collection of sites? Do certain streaming media not work? Come on, tell us what is wrong, so we can try to give you some proposed solutions.
Complaining that you disagree with the decision of your ISP is not the same thing as offering up a real description of the resulting issues that occur because of that decision.
I'm honestly very curious to hear, because I run such a proxy in my company's production networks, and sometimes my users are not actually the most vocal in telling my department if something is wrong. I'd like to hear what you're seeing.
Thanks in advance.
-db
Re:Err...so what is broken exactly? (Score:2)
The problem I'm having is that, because I have to rely on the proxy's DNS to resolve a web hostname, I can't get certain HTTP requests to certain hosts.
The specific example that I mentioned is the "http://www.dev.null" URL, but there are loads of other examples. In particular, "http://www.dev.null" is a completely valid URL that points to, well, the dev.null site. To resolve the URL into a host IP number to connect to, you have to be using the openNIC DNS tree (.null isn't supported by ICANN).
So far so good. I can connect my web client, for example, to "https://www.dev.null" and get the secure pages for the dev.null site. But the main pages at "http://www.dev.null" are invisible to me -- the most I can get is an error message from my ISP's ``transparent'' caching proxy.
The reason for the difference is that my home box uses OpenNIC -- the https: request gets resolved into an IP number and my local host just telnets to that host on the appropriate port and issues an encrypted HTTP request.
The http: request fails because, although my host resolves the host-part of the URL into an IP number, that number is ignored by my ISP's ``transparent'' proxy. All port-80 packets emanating from my home network get intercepted and fed into the proxy. The problem is that the proxy attempts to resolve the URL using ICANN DNS -- so it's unable to identify an IP number to associate with the URL. It's apparently poorly coded, in that it doesn't know enough to try the IP destination address on the incoming stream that it's intercepting.
My ISP reacted sort-of the way you just did: "well, if you're not using DNS, you're out of luck, pal." But that's Wrong. If they had an explicit proxy server and I had some choice about what happened to my packets, that would be OK -- but they don't. All of my port-80 packets are being intercepted by a piece of buggy 'ware. That (A) breaks the layered structure of the IP protocol, and (B) prevents me from accessing big chunks of webspace that I'd like to use.
If you administer a similar transparent proxy setup, I encourage you to ensure that it falls back gracefully to being a router if it can't make sense of the http: requests. Apparently the most popular Cisco solutions and Linux ipchains router/proxies are pretty easy to configure correctly (according to some of the other answers).
Cheers,
Craig
Re:Err...so what is broken exactly? (Score:2)
The proxy server, when operating in the role of an intercepting transparent proxy, should make the connection to the same IP address as the client did. And it should do it immediately if it has no cache entry for that (hostname,IPaddress) tuple. The proxy can then do a DNS lookup on that hostname later, to find out what IP addresses can have a merged/shared cache; later requests using one of those IP addresses would get the merged/shared cache, and those using other IP addresses would be cached separately.
So tell my why a proxy server can't do this.
Of course, it's not very likely the ISP will fix the problem very soon. Other solutions, such as an outside proxy, are likely to be the quicker workarounds. Ironically, this would defeat the bandwidth gains for the ISP, although for only an amount of that one customer. So the ISP shouldn't care as it's so small, right? If it turns out they do get angry that the customer is bypassing their cache proxy, then it's time for them to get their clue.
tunneling over other than ssh... (Score:2, Informative)
I have seen tunneling via ip-ip, ssh, and other ipv4 protocols mentioned, however there is another option available, and that is to tunnel your traffic as ipv6 traffic over ipv4.
It does take a bit of time to set up, but if you can find an agreable ipv6 network provider to allow you to tunnel to their server, your traffic will not be handled by any transparent proxy server at your local ISP, regardless of the type of traffic that you are working with.
I am not sure how complete the ipv6 implementation for Windows is yet, or, depending upon which version of Windows you may be running, if it is even an option, but for users working with Linux and BSD, this should not be a significant issue.
Then again, I could be wrong.
-Rusty
Yep (Score:3, Informative)
This is how a 'transparent proxy' is going to work. Any SYN to port 80, whether from telnet of a browser, is interecpeted. Then the proxy uses the URL and host in the GET request and headers to get the page and send it to the client. Your only hope is do your browsing on a port other than 80, as others have noted, by setting up a machine somewhere outside of your ISP that can recievie your requests on something other than 80, then send them out on 80. Then you set your browser to proxy to that other machine on that other port, if it's an actual proxy.
Of course the real cure is to shit-can your ISP. Not only are they messing with your abiity to use the DNS root of your choice, but they also have the ability to track your activities. (well, more easily than just putting a sniffer on.) Wonder what they're doing on port 443?
Nope (Score:2)
There are 2 different kinds of proxy roles.
One is where the client browser connects directly to the proxy. It provides the hostname or IP address in a URL on the GET (or other method) request line. That's the one to be used.
The other is the interception of TCP traffic (usually port 80 specifically). In this case, the browser is unaware, and is connecting to a specific IP address. But most importantly, it is sending a URI, not a full URL, on the GET (or other method) request line.
At the very least, a proxy which wants to connect to IP addresses it gets from DNS instead of the one the client was connecting to, should, if DNS gives it no addresses, connect to that IP address the client was using. The proxy peakpeak.com is using can't even accomplish this. Now that's just dumb.
If it were to use the proper origin server IP address and the associated name from the HTTP "Host" header to first check the cache, then connect before doing a DNS lookup to that IP address, then performance is faster. There is no need to do a DNS lookup for this purpose (although it can be later used to know a set of IP addresses that can share a common cache).
send it some garbage (Score:2)
My ISP (CTC) started doing this on my static dialup without warning. I noticed because 1) eBay pages suddenly required reloading in order to update (ie, if I quit the browser, and then went back to a dynamic eBay page, it was the same as before unless I reloaded the page). .. and then 2) I noticed when connecting to another machine, the address that showed up in the logs was not mine!
Anyway, after poking the machine I discovered it was a Cisco something or other. I also discovered that if you sent a malformed or invalid request, it would STOP transparent proxying for a few minutes!
So the solution I came up with was to telnet port 80 someplace (didn't matter where, because the proxy would pick it up) and type "PLEASE DON'T PROXY ME" and close the connection and then it would leave me alone for a few minutes.
Most of the time I left it on as the proxying seemed to speed up the usual day-to-day surfing. But you might want to try a script to do this automatically. Probably this is just an option the engineers forgot to turn off (I believe by law they must turn off all customer-friendly services :-).
After a few weeks of doing this, and making a few phone calls, the proxy mysteriously went away. Maybe they took my static dialup off the list, or they decided to do it for everybody. Whatever. I've been using Squid so it's pointless for me anyway.
Re:send it some garbage (Score:2)
What makes you think it suspended proxying for all the other customers/user when just one customer/user does this?
Re:send it some garbage (Score:2)
Ha! 1 0wn3d 17! /\/\3 S0 L33T! LOLz!
Ahem. It didn't get DoS'd, it simply stopped intercepting packets from port 80 from my particular IP address for 5 minutes, not everybody else. It's a transparent proxy on outgoing port 80, there's no guarantee that port 80 traffic is HTTP. I sent it some non-HTTP traffic.
It also did a similar thing when I went to "http://blahblah.com" where blahblah.com is a host that isn't running a web server. The first time it would hang, subsequent attempts returned were "connection refused" directly from the remote server. (I assumed anyway, unless some really weird fuckery was going on).
Sending non-HTTP traffic out of my port 80 should not screw up the transparent proxy. How about if a client dies in the middle of a request? (Sending "GE" instead of "GET /" or whatever). Or what if they want to connect to a MUD on port 8080 and mistype 80 or something. Or what if somebody is running an SSL server on port 80. That's one brittle piece of firmware if the whole show goes down because of that! Hackers would love it and embed https://host.com:80/ images in all their web pages to crash all these proxies.
Re:send it some garbage (Score:2)
Actually (for the benefit of anybody actually reading this thread) it would only suspend proxying based on REMOTE address. i.e., if I went to playboy.com port 80 and typed garbage, then I could subsequently connect directly to playboy.com:80. If I then went to hustler.com:80, it would be via the proxy unless I did the same trick. I couldn't test to see if it affected playboy.com:80 for EVERYBODY or not.
Now I recall why I never wrote a script to do it automatically....it had to be done for each remote addr.
http / 1.0 ? (Score:2)
you could also setup a proxy on localhost that rewrites the Host header from 'Host: www.weird_ass.domain' to 'Host: www.weird_ass.domain.existing_domain.com', and then have the DNS server that resolves 'existing_domain.com' to reply with the IP for 'www.weird_ass.domain' when it gets a request for 'www.weird_ass.domain.existing_domain.com'. Maybe the maintainers of the 'weird_ass.domain' zone alredy have that.
You'll probably need a lot of custom code for something that can be fixed by changing ISPs tho.
How this is done (Score:2)
It's working correctly (Score:3, Informative)
I see what you mean. You are sending traffic to a particular address based on your own DNS resolution, and if the traffic is proxied, you want it to be sent to your chosen destination, not that of the proxy.
In my opinion, the ISP is exhibiting correct behaviour.
Picture this: the object of the exercise with the transparent proxy is to cache pages and increase speed for the customer, right? I think it's already been agreed earlier in the thread that this is not entirely evil.
Let's say the proxy honours the destination IP address that you chose (I'm not sure how this would work in practice, but I'll go with it for now). It returns the web page from the server that your DNS picked, and caches it for the next guy.
Another customer requests a page with the same name. What if they're using a DNS root where the answer conflicts with yours? The customer gets the "wrong" web page. Because cached objects eventually expire, this means that the customer might get a completely different site dependent only on the time and date they happened request it.
The ISP doesn't use the same DNS root you do, so they can't begin to troubleshoot the problem.
I concede that the popular "alternate" DNS roots have few enough conflicts with the IANA-assigned roots at the minute, but even that is an irrelevancy - any solution that allows a customer to choose destination IP address on behalf of other customers opens up the ISP to a denial of service attack by a user less trustworthy than you or I. One could set up an arbitrary "root" server that resolves www.yahoo.com to my own site. Or google. Or some site that accepts credit card orders.
I can't see any scalable way out of this without the ISP picking one root, and sticking with it. If that is so, then I think this is a fundamental problem with split roots and, if you really want to use them, be fully aware of what you're getting yourself into. Turning off the transparent proxy will help this time, but you won't be able to rely on being able to talk to any server on the internet that doesn't use the same root as yours, even the servers you don't (usually) need to know exist.
Regards,
Dave
OpenNIC (Score:3, Informative)
His own box is properly configured to do OpenNIC lookups, but the HTTP request to the (proper) webserver gets intercepted. Now the proxy has to do the real HTTP request, but the proxy does not know about the alternative domains and probably returns a "Host not found" error.
I haven't heard of free proxy servers supporting one of the alternative NICs and I doubt the ISP will be interesting in subscribing to such a service. I guess the only solution will be to convince a friend to set up a proxy on a box someplace else.
Some alternative roots have their own "real" Internet domain which acts as a gateway domain, for instance name.space has http://name.space.xs2.net/ (regular hostname) which enables non-subscribers to view http://name.space/ (namespace only), making the domains available globally. If OpenNIC provides such a service, an alternative solution could be to run some proxy at home and let it rewrite OpenNIC urls into "regular" URLs.
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
Cheers,
Craig
Re:Wasn't port 80 supposed to be HTTP? (Score:5, Insightful)
The real problem is that you're probably using port 80 for something other than what it's explicit purpose.
No, that's not it at all. Follow the openNIC [unrated.net] link.
What he's trying to do is resolve an address, via the perfectly standard and normal DNS protocol, with an alternative root server. This is also perfectly standard and normal. This is not a violation of DNS, nor any other protocol, nor is it a particularly wierd thing to want to do. (Unusual, but perfectly normal.)
The problem is that his ISP is catching all traffic to port 80, and redirecting it to their proxy. Thus, when he asks for "http://www.something.nonstandardroot", the web proxy is interfering with the request (presumably after his home computer correctly resolved the DNS address of www.something.nonstandardroot), catching the GET part of the HTTP request, extracting the server name, and attempting on it's own to resolve the name.
(Note this is a complete waste: The home computer has probably already resolved the address, now the proxy will resolve it again.)
Unfortunately, the proxy is too ignorant to know how to resolve the alternate DNS address. It's not incapable in the technical sense, it just doesn't understand root servers it's not configured for. The problem is that this means that the perfectly normal and acceptable HTTP request, for an HTML document, on an IP address the client computer has already perfectly normally resolved, gets lost, because the proxy doesn't know how to resolve the address. Bad proxy!
A workaround, albiet a sucky one, is to resolve the address on one's home computer, then go to that IP address manually. This still causes problems on subdomain-aware webservers, where several domains or subdomains may all come from the same IP address, and the server wants to use the host part of the HTTP GET request to differentiate what to serve. (You could code up a quick Python/TK script to do this, but it'll still suck.)
So, when you say a proxy is not required to route anything anywhere, you've accidentally hit on the exact problem: a proxy shouldn't be routing, because it may not know how. This proxy tries to. That's why it sucks.
And to cover the last part of your post, there's absolutely nothing non-standard about any of this, except the behavior of the proxy, which is the only thing in this whole mess that hasn't "embrace[d] the DNS standard, HTTP standard and the routing standard". ICANN's root servers are not written into RFC's. They are merely common practice, one that many people, probably correctly, believe is an increasingly dangerous common practice [kuro5hin.org]. (You may not completely agree, but the opinions deserve consideration.)
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
http://1075594134 [1075594134]
Biggest problem with proxies is sites like slashdot with updated content. GFX can be cached, but the html pages are updated every time. Sometimes I find it funny at work, when a site has the wrong time cause its cache. I force a reload that fixes the problem.
At work we have a caching/compression solution, we can speed up text almost to 768K+ on wireless connections. Gfx are compressed, but not as much as text. Just imagine GPRS connection with a good caching/compression solution, DSL type speeds. (shameless plug) You can get it at ATTWS [attws.com] with Pocket PC.
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
*sigh*.
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
The first two solutions are obvious, and I'm surprised caching proxies still don't use either of them. I'm sure they've been suggested before, but I haven't been keeping up with the caching IETF working groups.
Solution 1:
Send to the proxy the address of the DNS server you want the proxy to use for resolution. This is a kludge, and would result in duplicate DNS queries, which can take a long time, but at least the proxy would see the world as your client would see it. Unfortunately, you're either breaking the proxy's transparent feature by doing this, or you're mixing up the layers and violating good architecture by embedding resolution information in your HTTP client request.
Solution 2:
HTTP/1.1 specifies the host field for HTTP client requests. The solution is to also provide an IP address field, but make it optional. If it is present, then the proxy would not resolve the host field to an address, but otherwise it would. It's simple and it does not violate the nice layers set up in the architecture.
This is the solution I would like to see, and it is so simple.
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
The proxy server is enhanced to try multiple DNS servers, (even in event of NOXDOMAIN), including some in the 'standard' tree and some from OpenNIC. This solution has the advantage of only needing mods to the proxy, not the clients.
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
This also avoids performing unneeded DNS lookups.
Re:Wasn't port 80 supposed to be HTTP? (Score:2, Insightful)
The problem he's having is that he's asking for an OpenNIC web site, and not receiving the page. The problem is as follows:
The "address" of the site he's looking for is present in two separate places in the request he's making. The IP Header includes the IP address of the site, and the HTTP header includes the URL, which includes the server name.
When he requests a webpage from an OpenNIC TLD, his machine correctly resolves the hostname, and constructs an request, which is sent through his ISP. The web proxy intercepts the request, and tries to proxy his request, so that it can be cached for later lookups.
Apparently, the Web cache is not configured to lookup machines under OpenNIC TLDs. That's reasonable, but that shouldn't stop a web browser from being able to see the web page.
If the web proxy can't identify the hostname present in the URL, it should simply pass it through, allowing the client (who already knows the IP), and the Web Server (who also, clearly, already knows it's own IP) to communicate. This would prevent the client from gaining the benefit of the cache, but would allow the client and server to communicate.
By accusing the poster of "[choosing] to disregard the other relevant standards," I can only assume your talking about his testing the web requests through a telnet client. I think that was an excellent troubleshooting procedure. It clearly identified the source of the problem.
HTTP does have it's own rules, but none of those rules should override TCP/IP. If this user makes a request to a web server (he's obviously already identified the IP address of the server, or he wouldn't be attempting an HTTP request). The caching proxy shouldn't be hijacking his request for any reason. It may be misconfiguration, or it may be broken proxy software, but it certainly isn't the user's fault.
Re:Wasn't port 80 supposed to be HTTP? (Score:2)
Not a horrible idea, i'm fairly doubtful it will work though.
The problem is the caching architectures i've seen have the exit router communicating to the cache through a gre tunnel, so the cache has no way to actually let it "pass through" as the cache isnt the actual box doing the routing. This is vastly superior to "put proxy between person and internet and let it route too" because of loadbalancing and failover reasons, but doesnt allow what you're asking for.
I dont really consider this 'broken proxy' nor do i consider it a misconfiguration, the simple fact is that his ISP (as well as mine) denies the existance of the opentld system. If he really has a problem with it, he should find another provider. Expecting the ISP to bend over backwards to satisfy their
Re:Wasn't port 80 supposed to be HTTP? (Score:3, Insightful)
If you connect to a specific IP address, a transparent proxy should connect to that very same IP address. If it connects to any other for any reason, it is apply a sort of "routing" logic. Apparently what happens is because the client includes an HTTP version 1.1 "Host" header, the proxy prefers to do a DNS lookup on the hostname given, and (if it finds it) connect there instead of the client's original destination IP address.
This is broken. If the proxy has a different idea of what domain names mean, it gets the wrong web site, or perhaps fails to get one at all. A correct transparent proxy implementation should always connect to the very same IP address the client tried to connect to without regard to the "Host" header (which must also be passed along). A DNS lookup can still be done to optimize the cache. If the destination IP address is in the list of A records from the DNS query, then it can simply be matched to the cache by name alone. However, if the IP address does not match any that DNS gets, then those pages can still be cached, but they must be cached under the tuple of both the destination IP address and the "Host" header name together (as this content can be different than any other for the same host name or the same IP address).
Maybe someone can provide a list of which transparent proxy cache programs do it wrong, and which do it right (as I have not examined these programs). I don't know if peakpeak.com will change out the software once they find something that does it right (or even make a configuration change if it turns out that's all that is needed). Ironically, if you find an outside proxy server which can do it right for you, you could connect directly to that service via a different TCP port and end up defeating the efforts of your ISP to save upstream bandwidth by caching.No, it's not. (Score:2)
A transparent proxy *does* break standards. You are no longer buying an internet connection, you are buying a filtered, proxied, mutilated internet connection.
That aside, this is not the issue the guy is having.
He's trying to use an alternative DNS system.. but the proxy is using it's own.. so he is hostage to what his ISP wants to resolve things to.
As for standards.. the STANDARD is to route IP traffic, not analyze it, mangle it off to a transparent proxy, and then send it onwards.
Re:long-term solution (Score:2)
More likely solutio is to configure your own proxy beyond the ISPs contorl. Also not easy, since most people don't have machines in that position, but your suggestion is strange enough by itself.
IPSEC wasn't ever meant to be used for oppurtunistic encryption applications (like https, ssh, etc), but to establish connections on a more long term basis that would be used for arbitrary protocols, not such common ones.
Re:long-term solution (Score:2)
I have, it is entirely practicable, with the right infrastructure support. However the probability that the destination supports IPSEC today is so small as to be negligible. And IPSEC would certainly not help someone who insisted on using an alternative root.
IPSEC wasn't ever meant to be used for oppurtunistic encryption applications (like https, ssh, etc), but to establish connections on a more long term basis that would be used for arbitrary protocols, not such common ones.
No, IPSEC was designed for exactly that case, it just happened to be deployed for VPN. IPSEC was started ten years ago, long before SSL was developed.
Re:Hold on here! (Score:4, Interesting)
If the user configured his browser to use a specific proxy, then I would agree with you regarding RFC2068. The client in essence is delegating DNS responsibility to the proxy server. However, what is happening here is called transparent proxy. There is no DNS delegation taking place. And RFC2068 requires that semantic transparency be preserved (although it does not seem to differentiate types of proxies). It says:
semantically transparentA cache behaves in a "semantically transparent" manner, with
respect to a particular response, when its use affects neither the
requesting client nor the origin server, except to improve
performance. When a cache is semantically transparent, the client
receives exactly the same response (except for hop-by-hop headers)
that it would have received had its request been handled directly
by the origin server.
In this case the origin server would have delivered a web page (I actually tried it [dev.null] and it works [ipal.org] fine for me), and so the proxy has the responsibility to deliver the same thing. In that, it seems, it failed.
Re:Administrative competence / certifications (Score:2)
Your rant, taken more as a statement of the lack of general competence is somewhat valid, but I just don't see the connection to this specific issue, other than obligitory karma whoring.
Whatever...
-buffy
Re:Sounds like Cisco's WCCP (Score:2)
Now the question is, when someone connects to an IP address, does the proxy, which intercepted the request, connection to that same IP address? Or does it do a DNS lookup and try to connect to what it thinks is the IP address? And if it does the latter, is it smart enough to fall back to the clients original destination IP address if it doesn't get one via DNS?
The classic proxy server, which is connected to directly by the browser client, does not know what IP address the client wanted, because it didn't try to connect to one. But in the case of transparent proxy, there is now the issue of the IP address that the client was connecting to. Have you verified what your proxy is doing?
Re:Port Blocking (Score:2)
A minimal outside web service reached via the domain name you want to use can redirect all requests to a slightly different domain name, along with a port number which won't be 80. That won't prevent them from deciding to block every incoming TCP connection not recognized by various protocol proxies (e.g. FTP), so this isn't a perfect solution. As the cat and mouse game continues, you may end up having to keep up a tunnel to an outside server, and run that tunnel through faked HTTP as well.
Re:Isn't it IP-based if OpenNIC already resolved? (Score:2)
A server running on a single IP address can serve multiple virtual hosts using the HTTP "Host" header to select among those configured. If you typed "http://64.28.67.150" then the client will send as one of the HTTP headers "Host: 64.28.67.150". The origin server (the one the ultimate has the requested document) uses that to select a virtual host, if so configured, or ignores it if not configured with virtual hosts. The proxy server, however, should respect the connecting IP address and use the resolved hostname only for cache matching purposes (so it can match a common cache among multiple IP addresses listed by DNS, if the destination IP is among them). So in the case of your question, using the IP address like that wouldn't make any difference. Where the difference lies in when you connect to a hostname in which your browser gets a different IP address than the proxy gets.
Re:Well, duh. (Score:2)
TCP/IP does not mandate the use of any specific root servers. This kind of thing doesn't even need to involve alternate DNS name spaces to have an impact. It's perfectly valid to connect to an IP address which is NOT listed in the A-records for a hostname which is also included on an HTTP "Host" header line. Yes, these are weird things, but they are valid internet protocol. If a business wants to assert that it is offering true internet service, then it needs to make it work exactly as if it were. If they can pull off doing a cache that ends up behaving to the end user exactly the same, great. But peakpeak.com didn't accomplish that. An intercepting transparent proxy server should always connect to the true origin server, and pass any "Host" headers through unchanged. It must use them to check for cache validly, along with the IP address. Even in the case of peakpeak.com's broken proxy, which was wanting to connect to whatever IP it got from DNS, should still fallback to the IP address the client was using if it gets nothing from DNS. It didn't even do that. That's very broken.