Large-Scale Paper-To-Digital Conversion? 459
An anonymous reader writes "I've just been asked to digitize several dozen sets of lecture outlines at the university where I work. Basically, professors want to hand me a big (often 100+ page) stack of their handwritten lecture notes (with messy text, equations, and diagrams; sometimes double-sided) and expect me to post a PDF-or-something-similar to their course's web page. However, every desktop scanner I've ever used takes 1-2 minutes of user-attention per page and the resulting files end up Huge, impossible-to-read, or both. All I have at my disposal is my PowerBook, Acrobat, a couple hundred dollars of department funds for a new scanner (this maybe?), and, if I ask nicely, overnight use of the secretary's Win2k box. Any ideas? Sheet-fed scanner recommendations? Better file formats than PDF (or better PDF settings)? Do any of you students have usability advice?"
Get stuffed (Score:4, Insightful)
Re:Get stuffed (Score:5, Interesting)
I've had a similar job, where our school's lecturers wanted their notes in the same style so one of my jobs as admin assistant was retyping chapters from textbooks & inserting the original illustrations. That didn't start out too bad until lecturers started basing course notes on entire quarters of books, expecting them to be retyped completely in their own style. Give an inch they'll try to take a mile - use the few hundred $$ to get it professionally scanned.
Re:Get stuffed (Score:2, Interesting)
Isn't that copyright infringement?
Unless, of course, they wrote the textbooks.
Re:Get stuffed (Score:2)
probably
docutech is the way to go... (Score:4, Informative)
on a side note, if the professors are utilizing a lot of additional material which includes might include3 handwritten information, you might consider getting encouraging them to transcribe that material(hopefully your not the TA that has to do the transcription) into a digital for, be it text or WORD. this'll difinitely help in reducing the size of your files.
also consider looking into adobe's pdf service, [adobe.com] if you're overwhelmed with just orginizing the material itself. probably not so kosher to suggest ity on /. but it could be something the school already has an agreement with adobe(taking into account the units of acrobat the school itself might be using). i know it's not rolling your own, but sometimes using an "out of the box" solution to get thing up and running so you can explore other solutions has it's merit as well...
Re:Get stuffed (Score:5, Informative)
I've been working with various versions of the Docutech system for about six years, and they're in use in most of the professional copy/print shops around, at least in Scandinavia. They scan full page and double sided, 600 dpi at about 1 page/sec. Newer versions also can handle full colour.
Native document format is tiff images with a proprietary control file (structuring, positioning etc), but you can easily convert it to pdf.
I'd guess that a professional shop will charge you about 30 cents a page if you accept the raw document files without 'touching up'. This is more than adequate if you're just going to reproduce it on paper, or even distribute the PDFs. It'll weigh in at about 100k a page for the tiff format, and a lot less for the PDFs. This is black and white, which in most cases will suffice.
Professional equipment (as in contracting a print shop) is definitely the way to go. I know that at the University of Oslo, Norway, they have established an in-house shop that will do this type of work internally for just about cost. Maybe that's an idea to put forth to the management? Surely your university will find other uses for it than just your assignment.
Hope this helps
Re:Get stuffed - outsource to india (Score:3, Informative)
Ricoh Aficios, Ancient Fujitsus, and OmniPage Pro (Score:5, Informative)
we've gotten a bunch of jobs like this - turning handwritten documents into searchable pdfs
We had to do this, too. For a Court, which requires the reasons, decisions, etc. to be publicly available online.
*Thousands* of documents, hundreds of pages each. The responsible department got me, as the IT guy, to set it up for them (after they'd already bought the stuff to do it).
Basically, a couple of Ricoh Aficio series copier/scanners, a couple of ancient Fujitsu sheet-feed scanners, and a bunch of students sitting all day in front of computers running OmniPage Pro.
The Ricohs were great on paper - fast, networked, etc. but their scanner drivers were poor (reminded me of bad CD-ROM drivers - "Copywrite 1995 Behavior Tech Computer. All right reverse." [sic,sic,sic]), and their service (contract) involved having to call the Ricoh guy because the scanner portions randomly wouldn't appear on the network, then wait for him to appear while at least one of the students sat idle. 2 stars out of 5.
Ancient Fujitsu scanners, black and white only, don't remember the model number, required proprietary SCSI cards, no support under Windows NT/XP/2K. These were commercial-grade super-expensive scanners when new (about 1990). Installed Windows 95 on a bunch of relics with ISA slots for the SCSI cards and let 'er rip. Scanning was fast, feed was reliable like a good-quality photocopier or fax machine. Only issue was requirement for an old computer running an old OS; better overall than Ricohs - 4 stars out 5.
OmniPage Pro 12 - reading was *excellent*, far better than anything else I've ever seen. Handled French and English, simple monochrome diagrams, etc. with only very small occasional formatting problems. Print to a PDF using Acrobat on the file server. Only real problem was stability, frequently locking up and losing the scan and OCR on page 99 of a 104 page document. 2 stars out of 5, being punitive because of frustration.
As they got to be more proficient with OPP, and as OPP's dictionaries filled up, we were able to add more and more computers and scanners, so that they were running around, tossing files into the scanners, stapling scanned documents back together, and occasionally rebooting one of the Windows 95 workstations. Peak was 15 computers and scanners.
Task took 3 students 3 months full-time.
Re:Get stuffed (Score:3, Insightful)
Re:Get stuffed (Score:4, Funny)
Spend the money on buying them copies of Mavis Beacon Teaches Typing.
Re:Get stuffed (Score:3, Insightful)
How do you know he's getting paid to do it? Some professors have a nasty habit of getting all their nasty, menial and boring stuff done by their students who are already working on their degree projects 12 hours a day, six days a week.
Ok, so for some reason I assumed that the poster is a student so my initial reaction was probably off. I would never assign such a menial, dead-end ta
Re:Get stuffed (Score:5, Insightful)
I used to have a boss that would say things like "this should only take you about five minutes". I finally told him, "nothing takes just five minutes, if I have to stop what I'm doing there is a startup/teardown cost for every task." I convinced him that there was a granularity of 1/2 hour for every random task he wanted done. The discussion was fruitful for both of us, he was more reasonable about his expectations and put a bit more thought into what he wanted to distract me from my primary task to do.
Now, the original idea is a reasonable proposition, however, it isn't really the sort of thing that should be done for just one prof. Perhaps several departments can combine their resources to setup something that will allow this type of thing to done in a reasonable time frame.
plurvert
Re:Get stuffed (Score:3, Interesting)
Re:Get stuffed (Score:5, Insightful)
Besides the fact, it sounds like they are not aware of the time involved in scanning off 10's nonetheless hundreds of pages. It doesn't sound like they are too anxious to make it easy for him to get the job done either (not buying him new equipment, using the secretaries Win2k box after hours??).
I've volunteered my efforts before on a simple scanning job that required hundreds of regular photos to be scanned in at relatively good quality (why else do it otherwise), and ended up taking forever. Upon informing the client of the amount of time required, they adjusted the way the job was being handled.
I think being straight with your employers, and clients is the best approach to any situation where too much is being expected. The times I've had these instances come up, and recommended different approaches that resulted in money being saved, or manhours on a task being reduced, I saw benefit in my paycheck through raises or promotions.
Kinkos? (Score:5, Informative)
Re:Kinkos? (Score:5, Informative)
Our teachers use them for exactly the purpose described. If you don't have one of these type machines around anywhere, then definitely give Kinkos or some similar establishment a try.
Re:Kinkos? (Score:4, Informative)
If he's at a large university, some other department might have one of these. Xerox doesn't charge for scans when you lease the machine. They only charge for how many prints or copies are made, so it would be essentially free for another department to allow him use their machine. It doesn't even require any additional setup, since you can enter any email address into the machine and have it send the document there directly from the copier. (assuming the SMTP server has been set on the machine)
Re:Xerox Scanner doesn't do OCR (Score:3, Insightful)
There is no way you're going to solve that problem with one person and a couple hundred dollars.
I know there are Adobe archival systems that store the scanned image, along with whatever text they manage to recognize. You don't expect near 100% OCR accuracy from an old, largely handwritten sheaf of lecture notes and transparencies. But hopefully enough is recognized to be of some use.
Re:Xerox Scanner doesn't do OCR (Score:3, Insightful)
Works for me (Score:5, Interesting)
I tend to scan lots of documents and setup a simple perl script that uses the 'scanimage' command line tool to do the scanning. Using my Epson Perfection 1650 scanner (pretty standard flatbed scanner) I can scan an 8"x10" page in black & white mode in about 10 seconds.
I actually added a button to the Nautilus GUI shell so I can move to the directory I want and hit the button to scan a page to that directory. Very convenient.
I scan to tiff and then use the convert utility (part of imagemagick) to convert to png. The resulting files typically run about 100K to 200K depending on the content.
If anyone's interested in seeing the perl script I've posted it to: www.ollies.net/scanscript.html [freecache.org]
Steve
HP9200C? (Score:3, Informative)
Cheap? Dunno. It was just there. In any sort of volume though, the cost drops precipitously (cheaper that you doing a flatbed scanner!).
Check out something like that (or indeed that) used, use it, resell it. Or new, then use/resell. Or get the school to buy it.
If this is a continuous thing, then all the better to own.
8100C (Score:3, Informative)
Re:8100C (Score:3, Informative)
The color tiffs use a depreciated form of tiff that was rescinded from the standard as unworkable. On top of that, the version they use does not work with an variant of libTiff. Basically you are stuck working with a few windows programs... and graphics converter on the Mac. Photoshop with sometimes even choke on them.
When we try and scan yellow documents the scanner will occasionally freeze up. It seems to happ
Kinkos isn't worth it (probably). (Score:3, Interesting)
As other people pointed out, if you can get a couple of departments in on this, then you can more easily amortize the costs of really good equipment to do this...
One thing that I'll note is that I don't really like PDFs for this sort of st
well... (Score:5, Funny)
Plus, if you're lucky, you could also get other after-hours favors from the secretary as well
Re:well... (Score:3, Funny)
High Speed Scanner (Score:2, Informative)
Re:High Speed Scanner (Score:2)
Sadly, this approach is way out of league for the small budget the poster has.
I'd have to wonder if a consumer scanner, even a nice one like that HP, can keep up with the constant use required of it.
Much like Laser printe
Re:High Speed Scanner (Score:3, Informative)
I run paperport to store all of my bills, documents, etc. The HP scanner software simply will-not use the resolutions and options I want paperport to use (200 DPI, B&W).
When using the sheetfeeder, the damn thing always scans in 24bit at 200DPI no matter what I try and set as a default - then I have to manually convert every page.
Go with a different model.
N.
Simple. (Score:5, Funny)
Gotta be careful though. (Score:5, Funny)
"No, no, not my entire job, just this one part. No, I can do the rest. No, really. No! No... please..."
Re:Simple. (Score:5, Insightful)
Take the few hundred you have to spend on equipment and spend it hiring a few temps.
A good typist should be able to type up hand written notes faster than scanning them all in and manually fixing all the mistakes.
Re:Simple. (Score:2, Insightful)
Outsource the job to India.
Not as bad an idea as it sounds. My advice is to not waste the department's money, and your time, buying, installing, and using a sheet feed scanner. Somebody in your local area assuredly has one already that they either rent out to people in your situation, or that they use to do the work you need done.
Use the funds that the department gave you to have your local copy shop do the work. They will almost certainly do it faster than you could, and the end product will mo
HP Copiers (Score:3, Informative)
Re:HP Copiers (Score:3, Insightful)
I think the real answer is that this guy is S.O.L.
Re:HP Copiers (Score:2)
Re:HP Copiers (Score:2, Informative)
Re:HP Copiers (Score:4, Informative)
Worse case, you can get an HP scanner and the automatic document feeder for it. If this is going to happen a lot it should be pretty easy to justify the $500 or so for the scanner, ADF, and a copy of Acrobat.
HP Digital Sender (Score:5, Informative)
Re: (Score:2)
Re:HP Digital Sender (Score:4, Informative)
I've found the Canon Canoscan flatbeds do a good job of automatically scanning straight to PDF, only minimal user intervention (hit "enter") is required. There's a special mode for scanning text which enhances contrast, so messy notes and diagrams should be fine, too. The resulting PDF:s are also remarkably small in size for what is essentially a huge bitmap. I've a Canon Canoscan 8000F myself, it's very fast and can do higher DPI's than most people need, and although it might be a bit out of his price range, I'm sure the cheaper models can do the same job nearly as well.
Re:HP Digital Sender (Score:2)
Re:HP Digital Sender (Score:4, Informative)
I have one of these on my office network, and and I agree that they're pretty good machines--though I have some complaints about them.
First off, I don't believe their functionality justifies the $3100 price tag. While the feature set it good, for that kind of money, this thing should be able to OCR, and not have to rely on 3rd party software for that functionality.
Secondly, their "scan to file server" feature requires a server side daemon to run--you can't simply drop the document to an SMB or NFS server. Further, the daemon only runs on WinNT/2k/XP systems, and you need to do a little bit of hacking to get it to run as a service, instead of opening it manually (or via startup folder) on login.
Third, it can be DOG SLOW. In particular, when scanning multiple large jobs (particularly at higher resolution) the thing will bog down. It also can only handle a fairly small number of jobs in queue at any one time. One of our secretaries can fill its queue in short order, and have to wait about ten minutes before she can scan the next document packet. When she's trying to scan a hundred packets, this essentially becomes her main focus for a work day.
All in all, our Toshiba copiers seem to do the same job better--of course, they have their own problems (i.e. over $20k each, with a poor user interface, and they don't do color, and don't OCR either.)
Format (Score:3, Interesting)
Are there any web-based packages for searching documents, based on OCR-extracted keywords? Obviously with messy hand-written notes, formulas, etc, OCR won't work reliably. For a similar project, I'd like to OCR the files and use the text data solely for keyword searching. Obviously not perfect, but better than just images.
PNG is your friend....
Re:Format (Score:5, Informative)
The program I use to convert to TIF is IrfanView (http://www.irfanview.com/), a generally excellent image viewer. I'ts free, too, so no worries there. It offers a ton of options for compression settings for different formats, so you can try other file formats as needed.
Re:Format (Score:3, Informative)
I run a micro-publishing business which often involves scanning a lot of B&W images at high resolution. I'll agree that storing files as TIFFs makes them much easier to edit, though. Our final publishing happens as PDFs, though, and it does not bloat the size of
If you're being 'asked' (Score:5, Insightful)
It sounds to me like a damned hard job to automate (which is the only way it's not going to be a constant drain on your time), and you're being given next-to-no resources to even come up with a creative solution. Sometimes the best answer is in fact 'No' - it forces people to re-evaluate what they're asking. It comes with the danger of being sacked if it's you that's being unreasonable, of course....
Simon.
Re:If you're being 'asked' (Score:5, Insightful)
Explain the enormity of this scratched note-to-finished Pdf to this educator. Use crayons, mirrors, yarn and tape if necessary to get your point across. Just be diplomatic :P
Re:If you're being 'asked' (Score:2)
Re:If you're being 'asked' (Score:3, Insightful)
The most important thing (Score:5, Funny)
HP Digital Sender (Score:2)
http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/1
ADF Scanners (Score:5, Informative)
The scanner has four different pieces of software you can choose to use, I'd suggest Precision Scan Pro as that makes multi-document scanning easier.
Re:ADF Scanners (Score:3, Informative)
Re:ADF Scanners (Score:3, Informative)
Re:ADF Scanners (Score:3, Informative)
And for the record, you aren't limited to only 4 software applications for scanning (at least in Windows, any application will work if it uses TWAIN). Perhaps you were referring to the document feeder having limited software compatibility?
(Off topic, but amusin
HP Scanjet 5550c is not what you want (Score:5, Informative)
Our Engineering Society was trying to put up an exam archive with one of them and quickly gave up and started scanning with the flatbed.
Also the scanner has no sane support (one of the few HP scanners that doesn't)
DjVu (Score:4, Informative)
Re:DjVu (Score:5, Informative)
Re:DjVu (Score:3, Informative)
I haven't tried tic98 (mentioned lower in this thread) but I can vouch for DjVu. I routinely scan notices, bills and whatnot mailed to me, then destroy them (rather than maintain a large paper file)
300DPI Black & White scans take about 19kb. They are quite readable, and with 300DPI information, make pretty good printouts.
JBIG2 inside PDF (Score:3, Interesting)
So, I recommend scanning to TIFF (or TIFF inside PDF). Even if you don't currently have the encoding softeware, you can convert to
Fax machine (Score:5, Interesting)
Recruit the community (Score:5, Interesting)
Get several (dozen) other students to use their own equipment and time in echange for a copy/copies of the completed work.
I would hazard a guess that there are more than a few people who would like to have a copy of the complete series of the lecture outlines.
Easy (Score:5, Interesting)
a) Publication quality DVI/PS/PDF files
b) The student can deepen their knowledge of the topic
Everyone happy. Used to work like this at the university I went to. And you may be even lucky that some student typed these notes in for himself.
LyX for LaTeX!! (Score:3, Interesting)
Use the fairly user-friendly LyX [lyx.org] to do the LaTeX-ing.
Heck, get the academics themselves using it to prepare their notes in the first place!
They might actually thank you for introducing them to this convenient and easy document processor.
DjVu format is pretty good for scanned docs. (Score:3, Interesting)
There is an open-source project http://djvu.sourceforge.net/ that provides code for reading DjVu docs, but I have no idea where to get DjVu encoder.
PDF is good (Score:2)
We had a set of high-speed sheet fed scanners, it would be then checked, and linked to a database. The documents in most cases where shipped to a vault.
If you had a budget.... (Score:2)
They can scan direct to PDF at an amazing rate of feed using the standard sheet feed.
Since it has dual purposes, you might con them into one, shared among a couple of departments...
where to look (Score:3, Insightful)
But the broader question is whether this is really a good idea. The result is going to be huge files, which will be messy, hard to read, and will lack an index or table of contents. Seems like a case of profs with too much ego and not enough willingness to put their own work into more useful form.
Try making GIFs (Score:2, Informative)
Also, if you're scanning material with copy on both sides, you might get some visible bleed-through. Try scanning such pages with a sheet of black paper between the page and the lid of the scanner, then adjust contrast to ensure white whites and black blacks.
No good answers AFAIK (Score:5, Informative)
Electronic test-equipment manuals are pretty much worst-case candidates for scanning. In Tek's case, the schematic volumes often consist of hundreds of double-sided, nonstandard-sized foldout sheets (11x23" for example) with lots of fine detail that must be reproduced clearly. You can either scan the pages in segments and leave it to the reader to reassemble them, or you can take the manuals to Kinko's and have the foldout pages shrunk to 11x17" or 8.5x11" for scanning. Either way, it's a real hassle, and highlights a clear need for a "prosumer" duplex sheet-feed scanner solution.
A few years ago you could buy scanners like this one [ebay.com] that could handle arbitrary sheet sizes, but I haven't seen them in stores lately. These may be easier to use than flatbed scanners, assuming the precision they offer is sufficient for your application. I don't know how well they'd work on densely-printed schematics.
Other than bitching about the state of the scanner marketplace, I don't have much to suggest. There are a few hints that will improve the quality and usability of your final document:
Re:No good answers AFAIK (Score:4, Informative)
This camera can be controlled programatically. Automation would be needed to make it practical for a large scale, but it is much quicker than most flat-bed scanners and the quality would be okay for hand-written notes. It would be easy to take multiple overlapping pictures and leave it to software to re-assemble the images.
(Yes, it is a goofy solution, but I works well for me as I normally have my camera handy.)
Re:No good answers AFAIK (Score:3, Informative)
DjVu is probably the best format for the poster's needs. I had a university class where nothing was ever han
Digital Copy machine (Score:2)
If you can find access to a digital copier at your university somewhere, you can just put the whole stack of paper in the sheet feed and it should be able to scan every page double sided and put it on a network drive somewhere.
It might take awhile to figure o
Large Scale Paper to Digital Conversion (Score:4, Informative)
The problem is the mixture of graphics, equations, and text.
It's easy enough to turn a page of text into a smallish file. Get a good automatic-feed scanner ($3500 or so) and a copy of ABBYY OCR software. If the original isn't too speckly, tiny, or smudged, ABBYY will give you a 95% accurate text you can then correct. Best format to save in? Depends on what the school is going to do the files. If they're to be posted on web sites, perhaps XHTML. If it's just for preservation, plain text (if there's no Greek characters) or XML with UTF-8.
Equations -- well, there's supposedly a version of XML for math, but Distributed Proofreaders has ended up using TeX, as it seems to be the mathematical standard. While this would work for preservation, it wouldn't work for a web site.
For a web site, perhaps the best way would be to intersperse text with pngs of the equations and graphics. The pngs would still take a lot more space than text, but the files would be smaller than PDF versions of the whole page.
One solution (Score:2)
We bought an HP Scanjet with sheet feeder for about $200 (sorry, don't remember the exact model), and use Paperport to scan the documents to a network folder named for the person requesting the scan (the executive assistant does it). We save in 300 dpi TIFF files in 1 bit color (B+W), which are small (8.5" x 11" comes
A Fujitsu scanner, SANE and Quartz Python bindings (Score:5, Informative)
Unfortunately you don't have much use for something like Acrobat Capture because you have handwritten notes to deal with. To process the files, SANE [sane-project.org] and/or TWAIN interfaces are reasonably easy to write code for. The cool thing about SANE is that you can run the saned daemon on any Mac or Linux box, and with a couple of lines of config file changes, it's instantly available over the network from any Mac, Windows, or Unix box (there are TWAIN bridges for Mac [ellert.se]/Windows [ozuzo.net] so it even shows up in Photoshop and so forth); there are also standalone GUI clients like XSane [xsane.org].
I wrote a document management system in Python/wxWidgets (for Windows) in about a month part-time, and it works very well. Either on Mac or Windows, PDF makes sense because of the ubiquity of the viewers, even if you lose a bit in compression compared to more optimized formats such as DjVu. On Windows you can easily embed the Acrobat ActiveX control; on Mac OS X you have native PDF support, Panther's Preview kicks ass, and there are several open-source PDF browsing components such as the ones out of TeXShop or Glen Low's Graphviz port [pixelglow.com] you can embed in your own app.
Given a choice I would probably pick the Mac to do this project, because of the wonderful Quartz/CoreGraphics Python bindings. You can just draw right to PDF, and place PDF files as if they were images; for example, here's a short script to rotate a bunch of PDF files (sorry, Slashdot destroys Python indentation):
You could also use ReportLab [reportlab.org], but because a lot of the PDF processing code is written in Python it's somewhat slower and memory-hogging for high-volume use. (I used ReportLab on Windows for the above project, and use CoreGraphics Python bindings for my research, so I do know what I'm talking about mostlyRe: (Score:2)
My dad's office (Score:5, Informative)
he has a couple of high speed scanners from panasonic. They cost less than a thousand dollars (4-500) if I remember correctly, they scan at about 20 ppm, and the software that came with them will save each scanned group of pages as a separate document (pdf, tif, whatever). My dad uses this setup to scan all of the files that his cases generate (shrinking his document storage from about 1000 sq ft to 2 shelves in a bookcase). we are talking files that consist of 10,000+ pages, and normally he saves a years worth of cases on 3-4 cds. They can scan up to 500 pages at a time.
Here is a link:
High Speed Scanners [panasonic.co.jp]
All you can do... (Score:5, Insightful)
DON'T CLEAN UP THE SCANS. Don't even look at the scans. DO NOT RETYPE ANYTHING.
With the kind of volume you say you're receiving, the only way you're going to survive is to:
1. close your eyes,
2. load the documents into the feeder,
3. press 'scan'.
4. Make sure everyone knows this policy.
OR (Score:4, Interesting)
It's not a technology problem it's a problem of (Score:3, Insightful)
What I Use for a Similar Task (Score:2)
The scanner I use is an HP ScanJet
Some photocopiers support this (Score:5, Informative)
Re:Some photocopiers support this (Score:3, Interesting)
Comment removed (Score:4, Insightful)
University doesn't already have this service? (Score:3, Informative)
It takes away the cost of printing lectures/notes/required readings from the departments and tacks it onto the students who now seem to pay for printing above a certain limit in the labs.
At least this is the way at the universities I have worked at.
professionally (Score:3, Insightful)
The professional approach is to go back to them and clarify the outcome:
(a) you can scan the documents in, and they'll take X amount of space, and Y time; and this doesn't include OCR;
(b) you did a few tests (using the supplied document) and these are the results for TIFF, JPG, PDF, etc;
(c) OCR is probably infeasible (or not, do some tests) because of the nature of the documents;
Include in (a) the option of purchasing an automated document scanner, and the corresponding reduction in time.
Based upon all the above, get a clear go-ahead, and make the purchase if new equipment is authorised.
You said "where I work": this is your job: it's a bit poor to do as the other posters suggest and refuse to do the work: you need to make sure that the customer (professors) understand exactly what they are getting, and give them a choice to buy into it or not - i.e. "clarify the expectations".
If you assess that it's 2 weeks worth of work, and the professors don't disagree, then you're supervisor just has to put up with it.
PDF Settings (Score:3, Informative)
The you should scan in grey-scale or if there is high enough contrast (pen notes, not pencil) in Black and White. The grey-scale with a JPEG medium or even low compressions is going to be much smaller then the deafaults. A pure black and white with group four compression will be even better. At work we scan pages at 300 DPI that way and get 20 to 30 k files (I think, haven't done it for a while).
Also typically images for web viewing of even text are scanned at 72 dpi (all the scholarly journals at my university). This can make things hard to read but really shrinks the file (about 1/16th the size of 300 dpi).
Also if the scanner is set low res pure black and white it will scan a lot faster, but still be pretty slow.
The other option is to pay someone to do it. If you have all of the stuff ready at once and give the pros a week or so to do it when they aren't busy you can probably get as low as 50 cents a page.
Blah blah, I lost my train of thought 2 paragraphs ago
PDF of handwritten notes is DUMB!!! (Score:5, Insightful)
It sounds like the school wants to shift the production costs (i.e printing) to the students. This seems inefficient because the old way where the instructor could go to the copy center and have the notes copied the at the schools expense (I know these expenses are often passed along to the students anyway), rather than at the students DIRECT expense of their time for downloading, then printing out on their own equipment or using their own printing accounts at the computer center.
If the notes were being OCR'd and then made available on-line, or post processed in such a fashion (where they are searchable, indexed, etc) where they were searchable, it would be useful. Otherwise this seems like a waste of time and money.
-MS2k
Get a Canon Document Scanner (Score:3, Insightful)
The company I work at scans large amounts of documents to PDF format on a daily basis. Depending on the volume some people do, we use either a Canon DR-3060 or DR-5020 document scanner. These will scan both sides of a page simultaneously, clean up the image (despeckle and deskew) and convert them into TIF or PDF all on the fly. They're fast too. Between 20 and 50 pages per minute. Only problem is that they're expensive.
For your budget, you may be able to afford the Canon DR-2080C [canon.com] which goes for around $600. It has all the features of the more expensive ones, but it's meant for smaller volumes like what you're dealing with. With that, you'd be able to scan 100 pages into a pdf document in around 5 minutes.
ask your professor to be precise ... (Score:3, Insightful)
I think what your professor wants is not a bitmapped copy of his handwritten notes or some vector curves that resembles such, but actually a typeset version of the lecture notes. If that is the case, assuming that his handwritten notes are sparse (and hopefully without diagrams, since it takes more time to mess around with them), you can definitely do a stack of 100 sheets in a week, or, as someone already suggested, hire some typists to help you out.
Depends how good you want to do it (Score:4, Informative)
If a digitized copy of the manuscripts will do for you, you can go the scan -> image enhancement -> OCR -> save to PDF way.
For scanning, you already got a lot of good comments how to automatise the scanning of dozens of scripts. If you lack these possibilities also a SCSI or USB desktop scanner should do the job (it's definitely less than 1 min per page), so you scan a script in 2 hours. No need to bother to outsource the job to India. Probably you can scan B/W and don't need greyscale or colors. I would scan handwritten scripts at 200 DPI and save the whole pictures in front of the OCRed text, so the user doesn't see the OCRed text and can only use it for selecting and copy&paste. It would be too much work to correct the OCRed text here. For machine written text I would use 300 dpi or more for better OCRing.
As image enhancement you only need to be able to automatically orient the page so that the text is horizontal. I don't remember if Acrobat does it, but for this job I would anyhow get a good OCR program.
As OCR program I recommend FineReader, but also Omnipage is ok. FineReader does better OCR than Omnipage and Acrobat. It also saves better to PDF (with retaining all of the paragraph structure) than Omnipage.
If you keep the image before the OCRed text in the PDF you can expect files of 10MB for 100 pages for B/W scan at 200 dpi. OCRing of machine written text has become incredibly accurate, so you can do real OCR there and throw away the bitmap picture. This of course gives much nicer output (and smaller filesize), but you need to spend a lot of time correcting the text. Here the best OCR program really pays off (you probably have a lot of words which are not in a dict, need custom dicts (does Acrobat have them?),...). A program with a single flaw (e.g. that recognized you formula as text, or code as paragraph text,...) will let you waste a lot of time correcting it on every second page.
How I Do It (Score:3, Informative)
Stay with your current flat-bed scanner. Do not waste money on a sheet-fed scanner. You do not have nearly enough money for a high-end Fujitsu or Bell & Howell sheet-fed scanner which will reliably get the job done without mechanically screwing up. The pros use high-end scanners because they never screw up and they go fast. Cheap sheet-fed scanners miss sheets or jam up too often to trust them with anything. Make a sign-up sheet for work-study or volunteer students in your academic department to sit down at your computer and scanner and scan the documents into the computer. Give them free pops and gummy bears (slur it so it sounds like "rum & beers") or something similar which won't transfer from fingers to documents. Just take a few minutes to set them up and show them what to do. Keep it simple. Let those empty minds waiting to be filled with knowledge (and beer) do the time consuming zombie work. You should focus your attention on how to put the files on the website.
The scan file format I use is Portable Network Graphics format or PNG format. On average, it compresses black and white graphics 20-25 percent smaller than the widely used GIF format. PNG format is also supported to a basic enough level to be displayed using MS Internet Explorer, Netscape, Mozilla, and other internet browsers.
I use free Xsane scanning software on a linux system to scan the documents. Xsane can be set to scan in line-art mode, also known as black and white mode. This software can also be set to save files directly to disk in PNG format and automatically change the file names using numerical iteration, i.e., file-01.png, file-02.png, file-03.png, etc. without the need for human intervention to change the file name each time. I use a 100 dpi scan resolution setting because documents do not need to look ultra-smooth; they just have to be legible. Anything beyond that is a waste of hard drive space. Using this resolution also means I do not have to spend time embedding the graphic file in html code to constrain its width so it can be viewed on the average 15", 800x600 resolution monitor. I just insert weblinks to the individual, one-page graphic files: "Page 1, 2, 3, 4,
I'm archiving stuff at my university (Score:4, Informative)
We have a dual-processor G4 and an Epson 1640XL large-format FireWire scanner [epson.com] with the optional auto document feeder. It's probably a bit out of your budget ($2899 + ~$1200 for the ADF) but it's awesome. It can scan at up to 1600dpi and the ADF can automatically duplex and scan both sides of the page. We're using OmniPage Pro X for OCR software.
Right now we're more concerned with scanning the documents and getting them online, so we haven't started OCR'ing everything yet. But the ADF is awesome. It can scan both sides of all 300+ pages of a yearbook automatically in about 2 1/2 hours.
The newspapers are a bit different. They're getting a bit fragile in their old age so we have to manually scan them. We scan them at 300dpi in full color, so the 12x18 pages are around 50MB per page. But the scanner takes less than a minute per page. It's impressive.
We use Photoshop's web gallery feature to generate the image galleries. Pretty simple really. Let me know if you have any questions.
I do this all the time (Score:3, Informative)
Yes there are scanners out there that can work for you. I have a Canon DR-5020 which we just feed it a ton of paper and come back in a few and it's done. It can scan VERY quickly. PDF format would work just fine as well. It's the best option especially since it's hand written notes as well.
If this is a requirement which is going to be on-going then you will have to pony up the money and spend a few thousand. If you're not ready to do that, you may be in luck. Some places will lease it out to you and with that few hundred bucks I'm sure you can easily get a hold of one for about a week or 2.
Look up for people who do Document Imaging, and you should find a lot of business that come up. If you're in the washington dc area then maybe I can help you out quite a bit.
Don't bother (Score:3, Insightful)
The mere fact that it's handwritten means that it's basically a rough draft that was hastily flung together. Send them back to him, and have him type them in and rework them until he figures they're worth recycling for next semester. The prof will save time in the long run, and the students will have something nice, clean, and organized to peruse.
Alternatives (Score:3, Informative)
Check to see if the department copy machine has scan functions... most built in the past few years do, even if they aren't used in most places for that. You'll get a decent sheet feeder and way faster scanning than most desktop sheet-fed scanners.
If you have to buy something and have to go *really* cheap, you could get a multi-function print / scan / fax thing. Most will handle legal size, because they're not actually moving the sheet fed paper onto the flatbed glass... the image element stays stationary while the paper goes by. But, of course, you get what you pay for... expect to spend time dealing with paperjams and skipped pages. However, it should be faster than hand-feeding a flatbed.
Software:
I mention this simply because nobody else has (that I've found): Scansoft Omnipage Pro is designed for highly repetitive, batch-oriented OCR. It has options for doing automated or hand-tweaked "area recognition" (separating text from graphics) and has the best proofreading UI I've seen... it flags "low confidence" recognitions automatically, and displays both it's best dictionary guesses and the actual scanned words. Not sure it will help much with hand-written work, but for printed material it works well.
Format: Your primary concern when looking for a destination file format should be longevity... will the files be readable 5 years from now? I've seen a number of people recommending highly efficient but obscure compression schemes, which are a terrible idea if you want the data to stick around. Saving a few bits doesn't do you much good if you can't figure out what they mean. I recommend that people scan to two formats, just for safety (Omnipage can do this automatically).
-R
Re:Knee to the grindstone... (Score:5, Insightful)
Faust7 is right about this one. Frankly, OCR is ok, but not great - on nice text on book-or-better paper. Handwritten notes? With equations? No. Not unless your profs have some damn fine handwriting and we all know that that is absolutely not the case.
My advice is the same as Faust7's with these additions: spend some of that money on a really nice keyboard, wrist-rest and/or maybe a nice monitor. You are going to be needing all three. If there are any left over funds, get some really nice tea. I suggest Twinnings English Breakfast or Prince of Wales, if you're going to go bagged.
Re:Outsource it (Score:5, Insightful)