Should You Break TOS Because Work Asks You? 680
An anonymous reader writes "My boss recently assigned me a project that was all his idea, with two basic flaws that would require me to break multiple web sites' Terms of Service (TOS). Part requires scraping most of the site, parsing the data and presenting it as our own without human intervention. While we're safe on copyright issues, clearly scraping like this is normally not allowed. At times it might also put a load on those sites. The other is, for lack of better words, a 'load balancing' part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as 'distributed' computing when in reality it's 'parasitic.'
My question is: am I wrong about the ethics? If I do need to walk, how best can I handle it without damaging my reputation and future employment opportunities?"
Re:You're Right, Of Course (Score:5, Interesting)
I say do it. but EMAIL your boss with your concerns and then continue.
when the shit hits the fan you have documentation to throw him under the bus hard and watch the wheels crush him.
Honestly It's all about CYA in the business world. If your boss tells you to do something unethical or wrong, document it every way you can and hold onto that so you can hand him over.
Why do you have any loyalty to them? they have none for you.
Complain, then just do it. (Score:1, Interesting)
Every time I'm told to do something stupid (which is probably about 10 times a day at this point), I carefully outline in writing why I think what they want me to do is stupid and/or illegal, present that to management, and then do it anyway. If you are a programmer, you are not employed to determine whether something is legal or ethical or not. However, if you see your company doing stuff like this, it might be a good idea to look for a new job. Bad management does not limit itself to any one area of idiocy, and you don't want to tie your survival to a business that behaves that way.
Re:You're Right, Of Course (Score:5, Interesting)
All that will happen is that the site in question will blacklist your scraping application. I work for a media organization, and we deal with this stuff all the time. It's far more cost efficient for us to simply whack the application than to try and track down the jokers. It's actually pretty trivial to nail an automated scraper: they're obvious on the logs.
So the few times I've had someone ask me to do this sort of scraping, my response is usually that sure, fine, it works, but it's very easy to spot on the logs, and the information is very likely to become unavailable at unpredictable intervals.
In the long run, it's usually pretty futile to scrape in the first place. When you're stealing content just to drive traffic, you tend to have a crappy site. The only time I ever did a professional scraping app that was "justified" and "legal", the victim was another business unit within the same corporation, and we had every right to the data that they "couldn't" compile for us.
Re:You're Right, Of Course (Score:3, Interesting)
Re:You're Right, Of Course (Score:5, Interesting)
Re:You're Right, Of Course (Score:5, Interesting)
Maybe not in real time, but once someone detected a scraper at a given IP, they could easily change their site to feed that IP fake data instead of blocking it.
If I were in the scrapee's position, I'd probably do that because it's the best way to attack the scraper. From order of least effort on the scrapee's part to most:
1) Blocking it makes it obvious to the scraper that they've been found out, and they'll work around it, then you'll need to block them again, on and on the cat-and-mouse game goes.
2) Feeding them mostly good data but with lots of inaccurate information scattered about is nearly impossible for them to detect until it has irreparably damaged their reputation and/or caused them to make bad decisions based on the data.
3) Suing them is a pain in the butt, even more effort than 2)
Re:Sigh (Score:5, Interesting)
It's only any good if the other party co-operates. The boss can easily phone you or walk up to you and say "Yes I want you to do it." and you have no record, and for many people this is their default mode of operation because that way no-one can pin anything on them. Unless they're singing their own praises, when everyone gets cc'd in.
I used to find it infuriating but fury gets you nowhere in the workplace.
Re:You're Right, Of Course (Score:5, Interesting)
I've also had similar requests in the past, and in both cases I did the work. I considered the request, decided they were ethical (even if somewhat unusual) and so did it. That's something you're going to have to figure out for yourself - whether you're going to do it or not.
I've been on the other side of the fence also...
If you're relying on data for commercial use, putting yourself in a position where you need that data is a risky thing...
I had a scraper once come after me. I caught them - as the previous poster pointed out, it's easy... I didn't block them. I captured and redirected their requests so I could control what they got and, well, sent them some information that made them look really, really stupid. They were angry, but there wasn't much they could do.
They were just enthusiasts - they had no business risk in their application suddenly failing.
Let your boss know the risk he is facing and then ask him if he really wants to risk being caught and shut down unexpectedly, or worse, finding someone has poisened his data.
It's just not good for business.
GrpA
Re:Who cares? (Score:3, Interesting)
I wrote a little script to search multiple cities in Craigslist, simply because they don't offer the function at any price. People can say I'm a jerk, but I really don't care because it saves me a lot of time.
Re:You're Right, Of Course (Score:4, Interesting)
Tread wicked carefully! [slashdot.org]
Chip Salzenberg got his ass burned back in 2005 by grumbling about his employer's ethics regarding screen scraping. I heard him speak at YAPC::NA in Toronto that year, and from what he was saying, they were able to take his every legitimate action (e.g. logging in remotely to work from home) and twist it in court into something less than legit (e.g. unauthorized access). It's their word against his, and they hold the access logs. Your best bet, if you want to make a stand about the morals, is get the hell away from there first.
Re:You're Right, Of Course (Score:5, Interesting)
I find this discussion yet annother interesting insight into the (lack of) ethics of some company bosses. I've often found to my surprise, the ethics of sales people, marketing people and bosses are at times very different from that of programmers and other workers in a company. Some time ago Slashdot discussed "Ethics in IT" and its interesting how it fits with this discussion. Here's the link, it gets interesting how much it fits this discussion, once you get to the part that discusses how some bosses lack of empathy towards others...
http://slashdot.org/comments.pl?sid=448546&cid=22377570 [slashdot.org]
Some bosses have contempt for other people, so considering doing this kind unethical business behaviour, is well within their usual thinking.
Re:You're Right, Of Course (Score:5, Interesting)
>>>The other is, for lack of better words, a "load balancing" part that requires using multiple free accounts instead of purchasing space and CPU time for less than $2,000 USD per month. The boss sees it as "distributed" computing when in reality it's "parasitic".
>>>
Can someone explain what this means? Multiple free accounts of what? Gmail? I'm confused.
Since scraping is detectable, I would follow this course of action:
- tell the boss you think "we'll get caught"
- if boss appears to want to fire you, then go ahead and do the action, but ask for him to put it in writing
- note on the order you think it's a bad idea; keep original for yourself and hand copy to boss
- write the program
-
- (optional)
- from your home computer (using an anonymous account), tell the website what your program does, and explain you would have been fired if you had not complied with your bosses' wishes, but feel it's unethical to scrap data.
- watch as Boss looks like fool when website with stolen bandwidth decides to bar his company's access
- if fired, hire lawyer and sue the company for unjustified dismissal
$ profit
Re:Sigh (Score:5, Interesting)
On a similar note, email is great for lifting others up. A few years ago, I got promoted to a mid-level grunt, and worked a lot more with other business units. In big organizations, units don't play nice with other units. On the few occasions, someone went out of their way to provide good service and actually be helpful, their boss got an email letting them know. Maybe two or three emails a year, but those folks are the ones who are going to get the bigger raise and have chances at promotions.
Unfortunately, 95% of all emails about people are complaints. Do your job well, and the best you can hope for is to be ignored. I have personally tried to reverse that now that I am a low-level manager. I can't change the world, but I can influence my dusty corner of it.
Re:You're Right, Of Course (Score:5, Interesting)
The example that would leap to my mind is a number of services that allow you to "map" an ip address to a geographic location...I use one of those for my job search homepage, and it only allows ~200 queries a day for the "free" account...It would be plenty useful to have as a free service (targeted advertising), and if you set up enough "free" accounts, you could use it that way.
Since I'm doing all my job searching away from where I'm currently living, I use mine to make sure that my job searching page always looks "under construction" for people who live where I live. My boss actually checks it occasionally, I guess to make sure I'm not trying to leave.
Re:Sigh (Score:3, Interesting)
No, not bullshit.
This is explained to all new hires. Clarifications are to be done face to face. Not over the phone, not over email.
Email is considered a completely permanent record (even if it's not actually permanent in practice) and would be the equivalent of having a politician speak to a journalist on the record.
ie. you don't ever do it unless you have to, and then, you clear it with the press office first.
Well, the company I'm talking about was exactly like that. (But not politicians and journalists.)
Re:You're Right, Of Course (Score:5, Interesting)
There are just so many things that could be done.
They're planning on taking data from some site and pumping it to others and they have _ZERO_ assurance that it's going to be good data and continue to be good data.
When you do stupid stuff like this, if you're not careful very bad things could happen (SQL injection, maybe even malware slipped in) and they could just go "nope not us", and while you could try to sue them it's pretty darn hard to prove since you requested the "bomb", and it only appears once and never appears again.
If you're lucky it's just going to be goatse/tubgirl.
If you're not, it could be a lot worse. Just imagine the BOFH thinking "What should I do today to them and their users" and rubbing his hands with glee.
Just slightly tampered data will be bad enough.
Re:Sigh (Score:1, Interesting)
Heh, you haven't worked in a company like this. You send an email to your boss that mentions a couple normal things and slips in "And as you directed, I'm scraping the site that says don't scrape them." Your boss sends back an email that says "Gee, you have a lot of incorrect assumptions in this mail, I'm scheduling a meeting for us to clear this up." Then he tells you to your face that you're doing it or you're not a team player -- it's OK if have any more communications problems, he'll just note it in your file for the next review.
If they don't want to leave a paper trail, they won't. It'll just be your word against theirs, and they're the experienced manager and you're the employee with the poor work history. Shit rolls downhill.
The only thing you can do with a workplace like this is leave. Tell your friends to avoid it. Give the BSA or whoever an anonymous call if you think it'll help.
Re:You're Right, Of Course (Score:2, Interesting)
The problem is that you can end up on the hook legally, if there are legal ramifications. You clearly knew it was wrong, and did it anyway.
Re:Sure fire way to NOT get hired anywhere else... (Score:3, Interesting)
You are an employee of a corporate: protected by its immunity.
Employees have no immunity from felony prosecution. Bang, period, end of sentence. There is no immunity anywhere in corporate life. There is indemnity, which is a separate issue, which protects you from civil lawsuit; it does not protect you in any way from felony charges.
If you commit a felony and the police come knocking, expect to get charged. The corporation won't be.
And if you really think that protecting your company from a lawsuit and reporting possibly felonious actions to the company's legal department will get you not hired anywhere else, you really need to spend more time in corporate America. This is the way you handle these things. You don't involve the police and you don't go to the press. You go to the corporation's own internal hierarchy and say "my manager is doing something aggressively stupid which, if discovered, will get me in a ton of trouble and expose the company to massive financial risk. Please make him stop."
What do you think Legal exists for? They're there to protect the company -- from internal threats as well as external ones.
Re:You're Right, Of Course (Score:3, Interesting)
Believe me, there are *many* people around who think it is just the height of comedy to fuck with people who are basically stealing their stuff anyway.
You say that like it's a bad thing. Now where did I put my cattlepro... I mean cable tester.
Scraping data is a last resort, not the first thing you try. Forget the ethics - the fact he's working for a company willing to be that insanely cheap and stupid in the first place should be a signal to run far, far away in itself.
Seconded. I used to think my managers were daft, then I started reading thedailywtf.com and I gained a much greater appreciation of exactly how bad things can actually be. From the description this guy gives, he's definitely dealing with someone well on his way to ending up on that site.
Re:You're Right, Of Course (Score:3, Interesting)
I have a site which I get paid very well to manage under contract. I have traps all over the place.
one of my traps is for email scrapers ( php script that loads about 5000 names randomly over 50 pages )
another page I love is my "crap" page, when an ip hit's 20 times in 2 minutes, it loads up and ask if you are a human or a computer, humans end up on the recaptcha page and on there merry surfing way, computers end up in crap section, which is 100% non real data at bargain prices and very specific key word phrases that I can track down. causes more harm, since it also loads porn pictures and goat se and a few other ugly images. it works rather evilly well.
got a few others, but that's the basics.
if you want to read more about bot blocking go to incredibill page, that will help you alot
Re:Who cares? (Score:3, Interesting)
What "contract"? No contract was mentioned.
Re:I would not recommend this... (Score:3, Interesting)
They may have questionable morality but the fact is that you have agreed to work for and being loyal to them, don't sink to their level. They might even have legal grounds to sue you if they find out since you clearly have willingly sabotaged their business.
Since when did going in to work require you to hang up your morals and ethics at the door? If your employer is doing something unethical, many would argue that you're obligated (morally) to blow the whistle on it, since to do otherwise allows people to profit from unethical examples - setting a bad precedent. If your employer is violating a contract, you'd call them out on it. And that's exactly what a Terms of Service agreement is - a contract specifying the terms by which you may use the other site.
Re:Redirecting content (Score:3, Interesting)
IE also refuses to render it. But when downloaded to desktop, I got a thumbnail, and I can open it in Paint.NET just fine (it's actually 0x808080 rather than 0x000000). Paint.NET swallows 3.2GB of memory when I do this. Good thing I have plenty of RAM and a 64-bit OS :)