Sanely Moving from Word to the Web?

FooAtWFU asks: "I have a job for a web site (no link for you, Slashdot hordes!). A lot of it is systems administration and development, but I have to routinely post content which comes from a myriad of other sources. Usually they are from academic users, come in Word format, and ultimately need to be posted in HTML. The problem is that Word has all sorts of tricks up its sleeve to throw off the font, layout, size, and so forth. To achieve any sort of visual consistency on the site these various formatting tags all need to be scrubbed, but even using other office suites with better HTML export (OpenOffice.Org) to do the dirty work, it's often easier to recreate the formatting by hand from a plain-text version than it is to clean up a sea of messy tags. Does anyone have any advice (or magical tools) to help me deal with this sort of tedious cleanup?"

Scrapping

[fembots (753724)]

Interestingly, I have a similar job on a website (no link for you too, Slashdot hordes!), here's what I do (I'm sure there are smarter ways):

1. Place all "to-process" documents in a specific folder in a webserver
2. Write a script to read those documents
3. Use Regex (and similar functions) to strip off and/or replace specific tags/wordings (similar to web scrapping technique).

Admittedly it was a tedious job at first to identify every possible template, however I'm amazed how predictable some documents are and once you get hold of such "blueprint", you can reformat documents to HTML/XML fairly easily.

Once the changes are done, I then preview them in a browser, and if everything's expected, I simply save the page and use it; If not, it's easy enough to make a few tweaks from the familar HTML environment.

Sounds like you should release on sourceforge

[arete (170676)]

So if there's only a few templates and they were a pain to work out, how about releasing your regex scripts to sourceforge or similar? Or posting here?

Re:Sounds like you should release on sourceforge

[by Anonymous Coward]

Since they were produced at work, the copyright on them is probably owned by the company and not by him.

Plus the templates are probably in-house templates and thus would be useless outside of the company.

Actually, an NDA probably doesn't matter.

[arete (170676)]

In all likelyhood an NDA doesn't cover obvious works like this - anything that could be reasonably discovered publicly. Doubtless he couldn't post the _documents_ that he converted.

However, I am also not willing to just assume that no company would ever consider letting someone sourceforge a script like this. It is 1) worth good advertising and 2) clearly not important enough to be worth selling. Release it in the company's name, or not depending on what they prefer.

At a minimum a lot of small companies

Re:Actually, an NDA probably doesn't matter.

[mrchaotica (681592)]

Doubtless he couldn't post the _documents_ that he converted.

You realize he was converting them for the purpose of putting them on a website, right? ; )

Re:Scrapping

[dougmc (70836)]

Use Regex (and similar functions) to strip off and/or replace specific tags/wordings (similar to web scrapping technique).

Of course, you really can't properly parse html just using regular expressions. You can get it right 90% of the time relatively quickly, and a day or so work will get you 5% more, but you could spend weeks trying to get that last 5% -- and never quite get it.

It's really better to use things that other people have made for parsing html. For example, if you use perl (and you shou

Re:Scrapping

[cloudmaster (10662)]

For simplifying documents, I've found HTML::TreeBuilder to be a handy module. Then you just have to write code to simplify HTML (throw away useless tags, merge adjacent tags, etc), rather than worrying about reformatting word docs.

PDF?

[night_flyer (453866)]

you can either hot link to the .doc themselves in a new window or convert to .pdf and do the same thing

Re:PDF?

[cloudmaster (10662)]

I hate you and your kind. Yes, hate. :)

Re:PDF?

[ImaLamer (260199)]

"Print to PDF" seems to be the function that would solve all of these problems, but so would any others. Think you *could* print to a TIFF, PDF, virtually any image type with a *nix Word compatible program - then you can scan the image and OCR it to plain text. Antiword (mentioned by another /.er: http://www.winfield.demon.nl/ [demon.nl]) can convert DOC to plain text... there are thousands of options.

However, if someone is getting the idea for another open source project to solve this dilema then I'd suggest somethin

Re:PDF?

[FooAtWFU (699187)]

Not quite what I'm looking for. Maybe I should clarify: I want to remove the nonessential formatting, while keep certain niceties (in particular, italics for the names of papers they reference, hyperlinks for footnotes, etc) and convert the rest into something simple and plain with just-the-basics of HTML, so I can then style it to match the other pages on the site. Many of these documents go to collections: encyclopedia articles, book reviews, abstracts of papers. If they don't look consistant, then people

Dreamweaver

[necro2607 (771790)]

I would suggest using Macromedia Dreamweaver... it's what we use where I work and essentially all of our content entry involves pasting in content from Word documents supplied by clients. Dreamweaver is pretty good for formatting and working with stylesheets.

Re:Dreamweaver

[fean (212516)]

in Dreamweaver, there's a command "Clean up MS Word HTML". Its made to clean up Word's crappy html, and does a pretty nice job of it.

Re:Dreamweaver

[by Anonymous Coward]

Also note that you have the ability to cut and paste formatted text from Word into the 'Design View' within dreamweaver and DW will automatically reformat the incoming text appropriately. In my brief test to make sure i wasnt talking out my a** i found it even supports word tables properly.
If you paste text into the Code view, DW removes the formatting completely and just uses the raw text.

Re:Dreamweaver

[drmike0099 (625308)]

The OP is correct: 1) Open Dreamweaver. 2) Commands > Clean Up Word HTML... 3) Rejoice

Re:Dreamweaver

[ozbon (99708)]

Yeah, I found that the "clean up word docs" did about 80% of the work, and then it was just a matter of a buttload of search/replace stuff in order to get it to finish the rest.

Worked pretty well, once I'd got the search/replace stuff sussed out.

Mind you, on a big word file you can think it's crashed when actually it's just doing lots of thinking...

Re:Dreamweaver

[necro2607 (771790)]

We use Dreamweaver exclusively for HTML/CSS programming and it doesn't create code that doesn't work - in fact the code it creates is very very compatible: I've never had to manually tweak code to obtain proper design and layout with the sites I've built.

Indeed we'd love to move to advanced CSS for page formatting but that's a big step right now - there are no professional WYSIWYG editors that have the sheer range and quality of features we need - Page templates, ability for clients to update the site later

Antiword

[alanp (179536)]

Try antiword, it's got a real decent HTML option.

Textism

[NoInfo (247461)]

Here's a tool I saw linked off of O'Reilly Radar once:

http://textism.com/wordcleaner/ [textism.com]

I used it once and it did a pretty decent job at preserving the tables. Yet if they're using anything odd like graphics or it's been incredibly tweaked, it probably won't be 100% perfect.

Re:Textism

[e**(i pi)-1 (462311)]

a standalone Perl script, I use daily is demoronizer [fourmilab.ch].

One suggestion

[Da Fokka (94074)]

You might consider a pack of monkeys and typewriters. They can ultimately reproduce Shakespeare so maybe, maybe they might be ablt to properly reformat the HTML gibberish Word produces.

Of course, you could also outsource to India but that's unethical to both the monkeys and the Americon economy.

Re:One suggestion

[by Anonymous Coward]

It's hard to find qualified monkeys - most of them already have jobs editing /. and cnn.com...

One Word...

[ScentCone (795499)]

..."Intern"

Re:One Word...

[Cerdic (904049)]

No, no, no...

Usually they are from academic users

It sounds like this might be a university environment. The correct answer should be grad students .

HTML Export

[electroniceric (468976)]

If you're using Office 2000, you can find the HTML filter here:

http://www.microsoft.com/downloads/details.aspx?Fa milyID=209ADBEE-3FBD-482C-83B0-96FB79B74DED&displa ylang=EN [microsoft.com]

I believe this functionality is built into later versions of Word.

Per the site, this produces simpler HTML with Office-specific tags removed. With that done, you could probably use a PERL script, and you might also try writing some Word macros or COM/VBA scripts that clean up the document from within Word.

Re:HTML Export

[Marxist Hacker 42 (638312)]

Whew- I hoped I didn't have to post this 40 comments down in the thread. Yes, Office 2000 has the above tool- and Office 2002 or 2003 has it on the Save As menu. The option you want is "Web Page (filtered)|*.html". I saw an interview once with somebody on the Word development team, and he claimed that the original Save As HTML was built for passing Word Documents over the web- and never meant to be read by human beings as a web page at all. Web Page (filtered) cuts out all the extra shyte that Save As HTML used to put in for managing version controled updates and changing the font every bloody character- and builds a real web page.

Dreamweaver

[SlashChick (544252)]

Check out Commands -> Clean Up Word HTML in Dreamweaver. it does a nice job of getting rid of extraneous tags. While you're at it, take a look at Commands -> Apply Source Formatting as well. This can be customized to your specifications in the preferences section, and automatically tabs out, adds newlines, and converts tags to lowercase where appropriate in the HTML document. Dreamweaver is the closest thing I know of to a program that "automatically" cleans up Word HTML.

Good luck!

HTML Tidy

[N8F8 (4562)]

Save the Word document as filtered HTML and pipe the HTML through HTML Tidy [sourceforge.net]. Nice clean HTML.

Tidy Flags

[N8F8 (4562)]

Almost forgot. The Tidy Docs [sourceforge.net] will tell you to select "--bare" and "--word-2000" and I also recommend "--output-xhtml" and "--indent".

no link for you, Slashdot hordes!

[SeanTobin (138474)]

Hmmm... sounds like a challenge to me. Let's see what we can dig up.

Step 1: Let's look at his user page [slashdot.org]

Ahh! He put in a website with his profile. Let's all go and check out http://fennec.homedns.org/ [homedns.org]

Hmm... looks like a personal page. Not too sure what to make of the comic. Anyway, let's move on to..

Step 2: Let's look at his author [wfu.edu] page. Some interesting stuff here, including three separate e-mail addresses (which I won't post here. You're welcome :)

A-ha! There is a link to his employer! It's Economic History Services [eh.net]. And what do you know... there are a significant number of pages (especially under abstracts and book reviews) that seem to come straight out of a word processor, only with extensive cleaning. A quick look at the source reveals something interesting. It's clean. Very clean. We're talking on the level of I-use-vim-for-my-webpage-editor clean. Nice job.

Anyway, it looks like it was done by hand. I'm not saying its not good work (quite to the contrary), but I can see your need for an automated solution.

Re:no link for you, Slashdot hordes!

[Dunbal (464142)]

Which only goes to show:

      There is NO WAY the slashdot effect can be avoided. Resistance is futile...

Re:no link for you, Slashdot hordes!

[FooAtWFU (699187)]

My SSH connection to my server still lives; I think my task was accomplished well enough. :)

Re:no link for you, Slashdot hordes!

[jalefkowit (101585)]

This just seems like a perfect opportunity for the application of a little common sense along with just a hint of courtesy.

You must be new here.

HTML Tidy

[John_Booty (149925)]

HTML Tidy has a special mode for cleaning up Word's crappy HTML export. HTML Tidy is a free command-line tool that is also embedded in a lot of popular HTML editors.

HTML Tidy:
http://tidy.sourceforge.net/ [sourceforge.net]
HTML Kit (great integration with HTML Tidy; it includes HTML Tidy so you can just grab HTML Kit without grabbing HTML Tidy)
http://www.chami.com/html-kit/ [chami.com]

Countless other editors integrate with HTML Tidy as well. Have fun and good luck!

Get it in PDF first.

[frostman (302143)]

I'm assuming you have the right to republish the Word documents. I'm also assuming you have no control over how many Word-specific formatting features are used by the authors.

What I would do in your shoes is set up a (mostly) automated system to convert the Word files to PDF. You can buy Acrobat or you can go with a third-party, printer-driver-style converter, but in the end you'll probably save more headaches just using Acrobat.

Once you have a document in PDF, you can use any of the numerous (free and commercial) tools to convert that to HTML, text, whatever - all much more reliably than from Word directly. It's not perfect, but it's probably the closest you'll get.

Plus, you can post the PDFs themselves for download in case someone wants them - and at least Google will still happily index your PDFs.

Yes, you'll probably have to live with some NT variant to get that part done (though it might work with OSX) - but it's most likely your fastest path to *quality* conversions.

Resign from your executive position

[Fastball (91927)]

What is it with executives and directors and their fixation with sending simple memos and messages via Word attachments in e-mails? Everybody else is on board with plain text (except some folks who are smitten with font coloring). Why can't the dolts at the top of the totem pole type in their mail client's editor and hit "Send?"

Re:Resign from your executive position

[dougmc (70836)]

Everybody else is on board with plain text

I don't know where you live/work, but out here in the real world, not everybody is on board with plain text. Not anymore.

I use mutt and fetchmail in a company of Exchange users. Almost every email I get at work now, from everybody, is in html. (Unless I sent it to myself.) I don't like it, but I deal with it. It's certainly easier to deal with it than to try and change everybody else.

I could change jobs, but over something as trivial as html emails? No. I like my job, I like the people I work with, so I just bend like the reed in the wind ...

Still, the executives are certainly worse about email ettiquette than most, and it's not just in this company -- everywhere I've worked I've found this to be the case. They don't include Subjects at all, or include useless ones like `message'. Some will type up a memo and send it as a .pdf file attachment, or worse as a .bmp file. They rarely trim anything when responding to a post -- they just top post away. (But many people do that ...)

Amen

[Quadraginta (902985)]

Jesus, tell me about it. I get 30kb attachments merely saying "Got your email, thanks!" with "thanks" done up in some odd curly red font and a six-line sig, not to mention the twenty-seven 8x10 colored glossy JPG attachments with circles and arrows and a paragraph on the back of each one...

Re:Resign from your executive position

[VGR (467274)]

You think that's bad?

I was given 61 screenshots (blithely dubbed "program requirements"), each its own Word document. Each containing only a (weirdly scaled) picture, of course.

61 Word documents.

fckeditor

[mixmasterjake (745969)]

fckeditor [fckeditor.com] is an in-browser WSYWIG. It has a "Paste from MS Word" button that actually strips out a lot of the unecessary baggage. I don't know how well it handles embedded images or tricky layouts, but for the basic stuff it works well.

The interface is similar to Word - maybe if you're lucky, you could get some of your content producers to use it.

HTML Tidy program

[Todd Knarr (15451)]

One program I've had luck with is the HTML Tidy program at http://www.w3.org/People/Raggett/tidy/ [w3.org]. It seems to clean up code (particularly from Word) quite a bit.

WordML - FO - XHTML/PDF

[room101 (236520)]

Using a modern version of Word, output in WordML (xml format). Use a XSL stylesheet [antennahouse.com] to convert the WordML to FO (formatting objects).

From there, do anything you want, like XHTML or PDF.

Or just go to XHTML from WordML with some stylesheet. XSL is teh cool!

Net-It is your magical tool

[netringer (319831)]

Net-It Central [netit.com] is the magical tool you were looking for. With that you can just point it at the file share with the Word Documents (and Excel and Power Point...) on it and see them indexed and cross linked on web pages. It'll update the content as the source docs change.

Oh, you mean non-commercial magical tools?

Pagify

[bckspc (172870)]

Pagify [backspace.com] is a perl script I wrote to do this for another job. It's basically a series of regular expressions that: 1. purges all the proprietary XML gunk from the HTML file you save from Word. 2. chops the file into smaller files wherever a Heading 1 appears 3. attaches endnotes as footnotes to the appropriate pages. It's GPL'd, so go nuts.

Try this....

[mormop (415983)]

Demoroniser is, in the author's own man pages words:

A Perl script which corrects incompatible HTML generated by Microsoft applications. [fourmilab.ch]

You can get it from the link in the same page. I must confess that I've not used it myself (don't use Office/Frontpage) but if it does what it says on the tin it should sort you out.

Dreamweaver MX 2004

[greymond (539980)]

1) Open Word
2) Select All -> Copy
3) Open Dreamweaver
4) File -> New Html Doc
5) Paste
6) Commands -> Clean up Word Html
7) Commands -> Apply Source Formatting (if you take the time to set the programs preferences to what you like)
8) Done
9) Drink beer
10) Sleep

Re:Change the Model?

[danheskett (178529)]

That's the best way. Really. You want the data to be in a structured format. Semantically structured if possible, but at least structured. Define a bunch of templates. Use a templating system like smarty or whatever to make it happen. Give your users a simple form - HTML, Windows, Java, whatever, that selects a template and reads in a list of fields from the template. Dynamically generate the form fields to be filled based on the template. Store the data. To generate a page start from the master re

Re:Use antiword

[cloudmaster (10662)]

He's obviously got access to Word, and saving as text in Word at least preserves most of the whitespace-based formatting. IIRC, antiword is mostly useful as a last-ditch effort to read a word doc, a step above piping the doc through "strings". :)

Re:DEMORONISER

[Kelson (129150)]

The Demoroniser was nice in its time, but it assumes the output should be 7-bit ASCII, or ISO Latin-1 at best.

The Unmoroniser is an updated version that handles Unicode properly and will do things like convert proprietary Windows-only curly quotes to the appropriate HTML4 entities instead of dropping them back to less accurate, typographically offensive straight quotes. Same with ligatures and other characters that the Demoronizer would munge instead of convert.

http://rheme.net/unmoroniser/ [rheme.net]

Yes!

[Kelson (129150)]

PDF is most suitable for documents that need to be printed with specific formatting.

For documents that are going to be viewed online, it's infinitely preferable to use a free-form format like HTML (was designed to be) that can adjust to varying monitor and window sizes.