The Replicator

Introduction

1. A Java utility that prepares the master files for uploading to a website by compacting the HTML (removing unnecessary white space), collecting the files by age and bundling them in compressed Zip format.
2. A Java WebStart Application that each client runs any time they wish to bring their copy of the files up to date. It visits the website, collects the new files, unpacks and decompresses them. The website files may optionally be protected by a login userid/password.

Trying the Replicator

Why Use The Replicator?

• Browse the website online. This can be slow, and for dial up users, ties up the phone and incurs Internet connect time charges. Some users don't have direct internet access.
• Download the entire site as a giant zip file for local browsing. This is extremely slow for dial up users. Even for those with high speed internet access, it can be slow. It is inefficient since you must download a giant zip even when only a few members have changed. If you download infrequently to save transmission time, then you have to put up with an out of date copy. The people who most need local browsing (those with dial up access) are the ones least capable of downloading the giant zip.
• Distribute the website on CD and send out copies by mail. There is the work of preparing the copies and mailing them out, plus the postage. Even before the CDs arrive, they are out of date.
• rsync is a very efficient way to mirror sets of files. Unfortunately, it requires a bureaucratic hassle to use, getting permission from the firewall people to use its special protocol and persuading your ISP to run the rsync server software on their machine.

rsync

The catches are:

• rsync requires you to run the rsync program on the webserver that contains the master copy. It can be difficult to persuade your ISP to let you do this. This is why I personally don't use rsync to keep my website in sync with my local hard disk master copy.
• rsync uses its own specialised raw socket protocol. You may have to make arrangements with the firewall people to tunnel through at each site it is used.
• rsync is a native mode program. You need a different version for different platforms. It may not be available for all your client's platforms.
• rsync is designed for Unix. It is a bit of a kludge to get it working on Windows.
• rsync is somewhat more complex to administer than the Replicator.

How To Operate The Replicator

After you have done a batch of changes that you want to propagate, start a bat file called PREPARE.BAT by clicking the item in your start menu or the desktop icon. If your machine can support it, you may alternately use the faster PREPJET.BAT that using a compiled version of the program.

PREPARE.BAT creates compressed summary bundle zips of files that have recently changed and uploads them to your website using a third party program such as NetLoad. Netload automatically removes deleted files from the website.

Normally you would upload your website both in compressed and expanded form, though the expanded form is optional.

At any time a client wants to get the freshest files, they simply click Replicator Update on a web page, or click Replicator on the start menu or a desktop icon. The rest is fully automatic. When it is done, the files will be sitting on their local hard disk decompressed, identical copies to the master. The client program automatically fetches only what is needed. It does not need the help of any third party software such as a browser or FTP program.

Features

• The Replicator is written in pure Java. This means it will run unaltered on any platform that supports Java.
• Many clients can fetch updates at once.
• Clients can fetch updates as frequently or infrequently as they like and all still works.
• If a client is disconnected, when the connection is reestablished later, the program will recover without any manual help.
• If a client's hard disk crashes, to get back in business, they need to install the Java JRE then click on the Replicator icon on your website to reinstall the software and update all the files from scratch.
• Clients automatically only fetch the newest ZIP files and changes they don't already have. There is minimal downloading of deadwood (deleted or replaced elements) or elements the client already has.
• Deleted files are automatically also deleted from the client disk.
• The Replicator works such that you can safely upload new versions even while clients are busy updating.
• New versions of the client software and auxiliary files automatically install themselves.
• When zip files are pruned of deadwood, they are automatically consolidated to aim for the desired optimally efficient target size.
• The Replicator depends on the file dates to detect changes and to decide which files need to be delivered to clients. If you introduce new files will old dates, they will be automatically replicated.
• The Replicator works with a simple HTTP server. It requires no special software on the webserver. It optionally works with password protected files. I suspect it would also work fine with HTTPS encrypted files, though I have not yet tested that.
• If you abort either the sender or the receiver at any time, it automatically gets itself back in sync the next time you run it.

Configuring

# replicator.properties files.
# Replicator Configuration file for Mindprod project.

# Summary of what this files says:
# You want to distribute nearly everything in
# E:\mindprod.  The Replicator will put the zips
# in E:\replicator-staging.  You will upload those
# files to http://mindprod.com/replicator.  The clients
# will download the zips to c:\mpstaging
# then decompress them to c:\mp

# This replicator.propreties files is used used
# to configure a set of files for efficient distribution via a website using
# multiple archive zip files to send just the changes.
# Each parameter that follows must be specified on one line, even for multiples.
# Everything is case sensitive,
# i.e. upper/lower case has to be precisely correct.
# You specify the broad rules of what files you want to distribute,
# then refine your choice with exceptions to the previous rules.

# Directory without trailing \ where ALL the
# master files to be distributed are kept.
# What is common to all filenames to be distributed, possibly just a drive.
# Use local platform names e.g. E: or E:\mindprod for Windows.
# This name is purely for the local
# Master copy, not directory on the website or directory at the clients'.
# Absolute directory name.
# use \ not /.
SENDER_BASE_DIR=e:\mindprod


# Directory without trailing \ where all the archive zip
# files to be distributed are kept.
# Use local platform names e.g. E:\mindprod for Windows.
# This name is purely for the local
# master copy, not for the website or the clients.
# Make sure this directory is NOT included in the trees,
# dirs or files to distribute!
# It is NOT relative to the SENDER_BASE_DIR.
# Absolute directory name.
# use \ not /.
SENDER_ZIP_STAGING_DIR=E:\replicator-staging


# Which directory trees (directory and all subdirectories)
# within the base directory tree will
# be distributed.  The name is relative to the base directory, i.e.
# The name must include the directory, but not
# the base. The name must not begin or end with a \,
# though it may contain embedded \. Spaces are ok.
# Do not surround filenames in quotes.
# Separate multiple directory tree names with commas.
# * means all trees in the base directory, and all files in the base directory
# use \ not /.
TREES_TO_DISTRIBUTE=*

# Which directory trees (directory and all subdirectories)
# within the base directory tree will not
# be distributed.
# Don't specify directory tree names, unless they would
# naturally be distributed by being part of TREES_TO_DISTRIBUTE
# above.
# The directory and all its subdirectories will
# be withheld, even if they appear in the list to be distributed.
# The name is relative to the base directory, i.e.
# The name must include the directory, but not
# the base. The name must not begin or end with a \,
# though it may contain embedded \. Spaces are ok.
# Do not surround filenames in quotes.
# Separate multiple directory tree names with commas.
# use \ not /.
TREES_TO_WITHHOLD=renney,addictions


# Which individual directories (directory without subdirectory)
# within the base directory tree will
# be distributed.
# Don't specify any directories which would naturally be distributed
# by being part of TREES_TO_DISTRIBUTE above. You do have to mention
# directories that would be excluded by TREES_TO_WITHHOLD above.
# The name is relative to the base directory, i.e.
# the name must include the directory, but not
# the base. The name must not begin or end with a \,
# though it may contain embedded \. Spaces are ok.
# Do not surround filenames in quotes.
# Separate multiple directory names with commas.
# * means all files in the base directory, not including all dirs and subdirs.
# use \ not /.
DIRS_TO_DISTRIBUTE=jgloss

# Which individual directories (directory without subdirectories)
# within the base directory tree will not
# be distributed.
# Don't specify directory  names, unless they would
# naturally be distributed by being part of TREES_TO_DISTRIBUTE
# above.
# The name is relative to the base directory, i.e.
# The name must include the directory, but not
# the base. The name must not begin or end with a \,
# though it may contain embedded \. Spaces are ok.
# Do not surround filenames in quotes.
# Separate multiple directory names with commas.
# * means all dirs in the base directory, and all files in the base directory,
# not including all subdirs of those dirs.
# * means all files in the base directory, not including all dirs and subdirs.
# use \ not /.
DIRS_TO_WITHHOLD=jgloss\include,include

# Which individual files (which otherwise would not be distributed)
# within the base directory tree will
# be distributed.
# Don't specify any files which would naturally be distributed
# by being part of TREES_TO_DISTRIBUTE or DIRS_TO_DISTRIBUTE above.
# You do have to mention files that would be excluded by TREES_TO_WITHHOLD
# or FILES_TO_WITHHOLD above.
# The name is relative to the base directory, i.e.
# the name must include the directory, but not
# the base. The name must not begin or end with a \,
# though it may contain embedded \. Spaces are ok.
# Do not surround filenames in quotes.
# Separate multiple filenames with commas.
# use \ not /.
FILES_TO_DISTRIBUTE=

# Which individual files that would otherwise be distributed
# within the base directory tree will not
# be distributed.
# Don't specify file names, unless they would
# naturally be distributed by being part of TREES_TO_DISTRIBUTE
# or DIRS_TO_DISTRIBUTE above.
# The name is relative to the base directory, i.e.
# The name must include the directory, but not
# the base. The name must not begin or end with a \,
# though it may contain embedded \. Spaces are ok.
# Do not surround filenames in quotes.
# Separate multiple filenames, with commas.
# use \ not /.
FILES_TO_WITHHOLD=zips\cmp1.zip,zips\cmp2.zip,zips\cmp3.zip,zips\cmp4.zip,zips\
cmp5.zip

# Which file extensions should be withheld from distribution, no matter what
# directory they are in, and no matter what previous rules say about these files.
EXTENSIONS_TO_WITHHOLD=log,zip,digest,nlx

# Largest file to distribute in bytes.
# This is to prevent you accidentally distrubuting some monster file.
# Suggest leaving at 10 megabytes.
# The largest possible setting is 2 gigabytes.
# Large files are not spread over several zips.
# Big files will be sent as one large zip.
LARGEST_FILE_TO_DISTRIBUTE=10000000

# We optimise under the presumption that most users will check for updates
# more frequently than this many days between checkins.
# Setting this number too large
# penalizes brand new users.
# Setting it too small penalises users who update infrequently.
LAG=5

# Approximate size in bytes of individual zip archive files distributed,
# prior to compression.
# usually 2 megabytes 2000000, no commas
# It might be set smaller if users have very flaky connections and can't
# reliably download files that big.
UNCOMPRESSED_IDEAL_ZIPSIZE=2000000

# Compression level 0 to 9.
# 0=uncompressed 9=as compressed as possible.
# Bigger number takes more time to prepare zips.
COMPRESSION_LEVEL=9

# 'distributed' if you want excess whitespace
# removed from any distributed *.html
# This makes the files smaller and faster to load.  They look the same
# on the browser screen, but the raw HTML source is harder to understand.
# 'original' if you want the original HTML compacted as well.
# 'none' if you want HTML files distributed as-is.
COMPACT_HTML=original

# true if you want files that have new dates, but have not really
# changed since the last distribution to be redated back to their
# original dates.  This saves redistributing files that have not
# actually changed content.  This is useful for generated index files
# which often generate to the same output as previously.
# This affects the file dates on the originals and well as the files
# distributed.
# false if you want redated, but unchanged files to be redistributed
# with the new dates.
# For more information on how untouch works see
# http://mindprod.com/untouch.html
UNTOUCH=true

# Command to upload files to the Internet.
# Needs fully qualitication and .exe suffix.
# e.g. E:\Program Files\netload\netload.exe
# or C:\WINNT\system32\cmd.exe /E:1900 /C upload.bat
# Leave empty (i.e UPLOAD=) to bypass automatic upload.
# Don't surround in quotes, even if it contains spaces.
# use \ in stead of /.
UPLOAD=E:\Program Files\netload\netload.exe


# Following properties control the default client configuration:

# Project name for title, just what it is we are distributing.
PROJECT_NAME=Mindprod.com Website

# Globally unique name, starting with website name in reverse e.g. com.mindprod.
# This name allows The Replicator to keep several
# projects on the same machine from
# interfering with one another.
UNIQUE_PROJECT_NAME=com.mindprod.replicator

# Directory without trailing \ where ALL the master files
# to be distributed are kept.
# Use local platform names e.g. E:\mindprod for Windows.
# This is the suggested default for the client.
# use \ not /.
SUGGESTED_RECEIVER_BASE_DIR=c:\mp

# Directory without trailing \ where all the incoming archive zip files are kept.
# Use local platform names e.g. E:\mindprod for Windows.
# This is the suggested default for the client.
# use \ not /.
SUGGESTED_RECEIVER_ZIP_STAGING_DIR=c:\mpstaging

# Where on the website the archive files are stored.
# without a trailing /. Must have a lead http://
# e.g. http://mindprod.com/replicator
# use / not \
WEBSITE_ZIP_URL=http://mindprod.com/replicator

# Used when the client does not have direct access to the internet.
# Where can it get the zip files from a LAN or his local machine
# downloaded by a relay version of the Replicator.
# If the source of the zip files for the client
# is directory on the current machine, you might code:
# file://localhost/X:/replicator
# If the source of the zip files is a directory
# on another machine on the LAN you might code:
# file://bigserver/Cdisk/replicator
# where Cdisk is the share name.
# use / not \.
SUGGESTED_LAN_ZIP_URL=file://server/sharedarea/mpstaging

# Do clients need a userid/passwords to access the files from the website?
# 'none' means no user id or password needed to access files on the website.
# 'basic' means use a simple undigested userid/password to access files.
AUTHENTICATION=none

# true if you want additional troubleshooting information dumped to the console.
# false if you want this debugging information suppressed.
DEBUG=true
# end
    

Client Use

On subsequent uses, all the client has to do is click OK without filling in any fields.

SneakerNet/LAN Use

The only tricky part is figuring out the URL for where the zip files are stored. Try this technique to discover it. Put a dummy temp.html file in the lan directory where the zip files and the manifest.ser are. Now try to open it with your browser from the machine runing the replicator receiver, using file open or whatever other techniques you have, e.g. drag and drop. When you finally get it viewed, you will see its URL on the top line. Use that as a model to create the URL for the LAN directory. If that URL does not work, convert any vertical bars in the URL to colons.

The URL will have the form file://machinename/sharename/directory Look in your Network Neighbourhood to discover the machinenames and sharenames. You can also assign the remote directory a local drive letter so that it appears as if if were on your local machine. Then the URL becomes file://localhost/X:/somedirectory

You can also test the Replicator echoing files back to the same machine, without uploading to a website, by using replicatorreceivertest.jnlp.

Dial Up Use

Web Server Requirements

• Must serve HTTP documents.
• Must have some way to upload documents, usually FTP, or SFTP.
• Must have MIME types configured correctly for various extensions: jar : application/java-archive, jardiff : application/x-java-archive-diff, jnlp : application/x-java-jnlp-file, zip : application/zip.
• Requires no Java support of any kind.
• Optionally it may protect the zip files from unauthorised access with a basic userid/password.

Master Station Requirements

• Must have Java JRE 1.4.2_02 installed which includes Java Web Start.
• Must have an Internet connection, preferably high speed.
• Must have firewall permission for HTTP downloads and FTP (or similar protocol) uploads.
• 256 MB or more of RAM recommended.
• Normally you would be Running Windows XP or Windows 2000, but any platform that supports the JRE will suffice.

Client Station Requirements

• Must have Java JRE 1.4.2_02 installed which includes Java Web Start.
• Must have an Internet connection, preferably high speed.
• Must have firewall permission for HTTP downloads.
• 256 MB or more of RAM recommended.
• Normally you would be Running Windows XP or Windows 2000, but any platform that supports the JRE will suffice which includes Windows 98.

Deploying

Creating Replicator Distribution CDs

Include everything in the SENDER_ZIP_STAGING_DIR, namely the ZIP files, a copy of the replicatorreceiver.jar, the JNLP files, freshness.ser, the setup.exe, the replicator.gif and replicator.ico files.

Do not include the files in the program directory such as replicatorsender.exe or replicatorsender.jar. Also possibly include a copy of the offline Java JRE also onto the root directory of the CD.

Make copies of the CD and send them out by mail. To install the software, just insert the CD in the CD ROM drive.

When you run setup from CD, it will install the client Java Web Start application software and unpack the distributed data files in the zips. The JRE is there just in case the client did not have a recent Java already installed, (which includes the Java Web Start runtime). All the client need to is insert the cd to invoke the autorun feature to install the program and datafiles. If that does not work, the client can jump start the process by going to a DOS box and typing

If your CDROM drive letter is not R: you can adjust it to be. Right Click My Computer | Manage | Device Manager Storage | Disk Management alternatively Settings | Control Panel | Administrative Tools | Computer Management | Storage | Disk Management

The client can then continue via website updates or lan updates or further CD updates.

You can also create CDs at remote stations by using the replicatorreceiverrelay.jnlp to download the zips from the website. Burn a cd consisting of the all the files in the relay receiver's zip staging directory.

Files

Cost

What Is Included

• Master station software to prepare the website for upload.
• Client station software to download any changes.
• Complete, commented source code in Java.
• The right to use this software on as many clients as you wish.
• Right to modify the software.
• Support to get the program configured, installed and functioning properly.
• All bugs fixed.

What Is Not Included

• Web hosting. All software and files live on your own servers.
• FTP upload software such as NetLoad.
• Thawte Digital Certificate in case a self-signed one is not acceptable to some clients. They cost $200.00 USD a year. One would handle all clients.
• Exclusive right to use the software.
• Right to resell the software to other parties.
• Right to give the software away to other parties.
• Remotely trouble shooting Java Web Start installations on client machines. This is best done by someone local. Once JWS generally is working then it becomes my problem to solve.

Trouble Shooting

All files and directories the replicator touches must not have leading or trailing spaces in their names. It is ok to have embedded spaces, even double spaces in names. If they have lead or trail spaces, the Replicator will abort telling you which file has to be renamed. It will pick up where it left off after you fix the problem and restart it.

Most problems getting started surround not configurging the replicator.properties file correctly. Check the system out by using the replicatorrecivertest.jnlp to download the zips and unpack them to a unique directory, and make sure the files you wanted included are included and no more. You can adjust the replicator.properties to correct any errors, and the Replicator will automatically adjust without having to start from scratch.

To start from scratch, run startover.bat. If that does not work, delete the *.ser files in the program files/replicator directory and all the files in the SENDER_ZIP_STAGING_DIR.

Most problems seem to occur during the upload of the zip files, which is not under the direct control of the Replicator. Check that the files in the SENDER_ZIP_STAGING_DIR match the website in size and date. If you see mismatches, you can try deleting the offending files from the website manually, then retry the prepare. If you use Netload, doing local refresh and site refresh will often help get it back in sync. Try deleting any *.nlx files in the SENDER_ZIP_STAGING_DIR and any netload.chk files in the WEBSITE_ZIP_URL directory on the website.

If problems seem to be related to Jet and its DLLs, you can revert to the HotSpot version which is slightly slower. You just use PREPJAR.BAT in preference to PREPJET.BAT. The Jet version only works in Windows.

To use the Replicator off net, you need to use the replicatorreceiverviarelay.jnlp to collect the zip files off the website and store them in a staging directory. You then have to copy those files to some place where they are accessible via a lan, perhaps using a CD as an intermediary. You then fire up replicatorreceivervialan.jnlp from the same directory as the zips. If this directory is the same as the one you configured in replicator.properties SUGGESTED_LAN_ZIP_URL, it should work fine. However, if it is different, you will have to manually edit the codebase parameter in the replicatorreceivervialan.jnlp file to make it match.

The other way to use the Replicator offnet is to use replicatorreceiverviarelay.jnlp to download the zips from the website and burn everything in the staging directory onto the root of a CD. From there you can install the CD just by inserting it into a drive. You can use the CD to initially install the files, and subsequent CDs to keep the files up to date. You can also use the web to keep the files up to date after an initial CD install. When you use replicatorreceivercdX.jnlp it will copy the files off cd it has not already got.

If you make a great many changes to your website, it is best to first upload those individual changes before running the replicator, rather than getting the upload phase of the replicator to upload those files and its own in the same batch. Large upload batches sometimes fail, and need to be run several times. Netload need to be have its cached directory manually cleared after each failure.

Possible Futures

• A built-in FTP upload program so that there is no more dependency on an external third party one like Netload or FTP Voyager.
• Encrypting the expanded files on the website and in the zip files to keep it confidential.
• If a client inadvertently deletes files, they are automatically restored on the next update. Sweep to discover accidentally lost or deleted files and individually recover them from the website. This requires maintaining the website in both compressend and expanded form.
• Digitally signing the zip files to detect corruption or tampering. The client jar file is digitally signed now.
• A generic hook to run a some user code on each file just before it is distributed, e.g. to ensure macros or includes are freshly expanded.
• Use of thumbdrives to keep the files in a permanently encrypted state, opened only for viewing.

Summary

• Finding a suitable upload tool that will delete files. Netload is not officially supported any more. Without such a tool, you will have to manually prune files from the website that are no longer on the master hard disk.
• Ensuring Java is installed and working properly on all the client stations.

home	Canadian Mind Products
	mindprod.com IP:[24.87.56.253]
	Your IP:[80.134.30.163]
	You are visitor number 1717.
	Please send errors, omissions and suggestions
	to improve this page to Roedy Green.
	You can get a fresh copy of this page from:	or possibly from your local J: drive mirror:
	http://mindprod.com/zips/java/replicator.html	J:\mindprod\zips\java\replicator.html