The Great Synchronization Thread

[csaba]

Hello Guys and Girls,

It has been obvious to us, that the most needed and asked-for feature in ForkLift is the possibility to synchronize folders.
We are now happy to announce that we have started development of our synch module.
As we want to make this the best damn synchronization tool that ever was, we are asking You, our Esteemed User, for input on what you would expect from such a tool. To this end, we are now launching this new thread in our user forum, and encourage you to make yourself heard.

What we are looking for most, is descriptions of the tasks where you need to use a synchronization tool; and your ideas about how you would expect to manage the workflow of these tasks. We would like this to be a very open and active discussion, one that we hope will result in a tool worthy of the general excellence of ForkLift.

We will hear out any ideas, however weird they may seem :) You are welcome to post screenshots, diagrams, filelists, links, anything useful to this discussion.

Thanks for contributing

BinaryNights

[Adrian]

A fully-fledged synch system sounds great, but I'd like to suggest a few really simple things that would be a big help for uploading to a remote sites.

Being able to specify a "local path" for a connection is great; additional options for the connection would be:
- turn on sync browsing automatically
- don't confirm overwrites on the remote server
- force the remote server into the right-hand pane, or the left - the point is it should be the same every time. I have inadvertently overwritten local files with remote ones because I had focus on the wrong pane when I opened the connection. Remember, both sides have the same files so look almost identical!

I guess what I'm getting at here is that while full automatic synching will be enormously useful at times, streamlining the process of uploading just a couple of files would be great too.

Thanks for Forklift. Loving it.

Adrian

[yevgeny]

* Syncing between two remote servers: Amazon S3 Euro & US buckets, different SSHs.
* 'One master, multiple slaves'.
* A scheduled synchronization (in background).
* applescriptability.

[buddha379]

To start, a quick aside: thank god for Forklift. I came from a Windows desktop. As such I can wisk files around in Windows Explorer like it's an extension of myself. I love my Mac, but Finder is an atrocious piece of crap. Having dual panes is the only way to go, if you have network shares, or multiple network shares like I do. Anyway, on to sync.

First off, not sure there is really a lot of need to reinvent the wheel here. Rsync is a fantastic program and can do two way synchronization (EDIT: no it doesn't, only unidirectional, see edit notes below, though you can hack a two way sync with -u option, which says don't update newer files, which allows you to do sync up and then sync down, in essence performing a bidirectional sync based on date). Most importantly it can do it over SSH, so communication is secure. I say just do a pretty front end to that and hide the complexity from the user. Only downside is that it can be a bit sluggish with lots of files, building its file list.

I really like the scheduled sync idea from Yevgeny.

As for Applescript, I'd be happy just to access it from bash with a nice simple command utility, that I could call in a shell script.

Sync should also have push, pull or bidirectional sync. Push to push up to a server. Pull to yank down. And bidirectional to make sure that only the latest in both directories are copied to both the local and remote server. Bidirectional is the most important. I have a CentOS server at my office that I sync to manually now. Would be a lot easier if it was push button.

Sync should also run as a separate process, perhaps as a daemon, that can be kept running in the background, even if the user closes Forklift. Of course the user should have the option to shut it down or daemonize it when he is closing Forklift, or he should be able to set it in preferences, i.e. "always run in the background" or "only run when Forklift is open" or "prompt me to choose," something like that.

Edit: I tested Unison which does bidirectional sync, but it's codebase is stale ( no longer updated ), not to mention it's written in some funky language called Camel, which may be a cool language, but I wouldn't know because I've never seen anyone use it! So you may need to reinvent the wheel here, unfortunately, unless you expand rsync or use the -u option.

Edit 2: I just went through the unison install on my Mac and on my Linux box. What a huge pain in the butt! And because it's written in the never-heard-of-it-before language "camel", I think using it as a potential backend is completely worthless.

Edit 3: There is an opensource program called Cyberduck, but from what I read in the forums Sync crashes it regularly and is horribly broken.

[cintra]

First off, not sure there is really a lot of need to reinvent the wheel here. Rsync is a fantastic program and can do two way synchronization. Mosty importantly it can do it over SSH, so communication is secure. I say just do a pretty front end to that and hide the complexity from the user. Only downside is that it can be a bit sluggish with lots of files, building its file list.

ref. the above, I was going to say that in addition to my usual SuperDuper and Time Capsule backups, I use arRsync to sync half a dozen special areas.

You should at least be able to match the simplicity and exceed the features of arRsync, which is a very simple front end to Rsync..

mvh

[edified]

I agree about using rsync. You'll benefit from its maturity and ubiquity. Just make sure to have a better interface than arRsync. It's good software but the GUI could use some serious rethinking.

[honzab]

Hi, first of all thanks for ForkLift. It's absolutely great.

Speaking of synchronization, I am used to use a commander called "Total Commander" on Windows, which I find so far unbeatable windows-tool. It supports synchronization of files using the synch view (see attachement of this post) which I find pretty cool.

So if you want to grab more inspiration for ForkLift's synch tool, try to look at http://ghisler.com where you can download a shareware version of TotalCommander.

Hope it helps a bit, because I would really love to see this feature in next version of ForkLift :)

EDIT1: It seems that the attachement got somehow thumbnailed, so here's link (http://images.brucek.cz/total_commander_sync.jpeg) for it.

[funtuva]

I Agree 100% with other posts that Total Commander is so far unbeatable wiws-tool. It supports synchronization of files using the synch view (see attachement of this post) which I find pretty cool.

The sync windows is elegant, simple and VERY fast. It offers all possibilities, copy left to rigth, viceversa, show singles, duplicates, etc... I have tried other Windows tools (Viceversa, etc...) and Mac (Syntevo´s Smart Synchronization is the best, although I do not need the line by line comparison nor the 3 way comparison...) , ChronoSync is bloated and messy, File Synchornization is simplistic, Synchronize Plus...more of the simplistic, Synk Standard
is amongs the best but it leaves you with hundreds of conflicts to resove manually, although the approach is on the rigth direction.

My point is why search any more: Total Commander has been around for more than a decade in the form of Windows Commander, etc.. it is a fantastic 2 pane file manager, and the sync modulo is hard to beat, so I would LOVE to see it come just the way it is into Forklift. I cannot think of improvements, really. It just works. So Please...........give us something like that!!

[rthrash]

* Syncing between two remote servers: Amazon S3 Euro & US buckets, different SSHs.
* 'One master, multiple slaves'.


Definitely these ... including being able to sync/mirror mixed protocols: SFTP <--> FTP, S3 <--> SFTP, etc. (and obviously local, too)

[George the Flea]

My most common sync activity is Mac -> FTP server while working on a website. Usually I'll do this while working with PHP (could use MAMP, but it's difficult to get MAMP set up just like the target server, and sometimes the work necessary to migrate an installed PHP program isn't worth the trouble).

Automatic syncing (watching a folder and pushing changes to the server without needing input) is a nice idea, but realistically I want control. Sometimes I'll save a file midway through, and uploading it at that point would just break things.

I completely agree with Adrian who asked for the panes to be locked in while working with mirrored directories. Accidentally overwriting a local filed while mirrored with a remote directory can be catastrophic.

What I'd ideally like to do is the first time I sync have the program ask me whether I'm uploading, downloading, or mirroring (or whatever terminology you use here; push/pull works just as well). After I've designated which direction I'm working in for the session, I'd like the program to maintain a visual indication of which files are going to be synced in the main window (little arrows, icons, color-coded backgrounds, whatever). There should also be some way to temporarily or pseudo-permanently exclude a file from syncing, and then I'd hit the sync button and have the changes run immediately. (Obviously some way to change the settings would be necessary, too; say if I wanted to go from mirroring to pushing).

What I would not want to see is something like Yummy FTP's workflow, where you click sync, adjust settings in a sheet, wait while it scans the directories, adjust/confirm which files are sent in another sheet, then finally see a sheet with a completion report (some steps of this can be disabled in the original settings sheet, but still). This takes far too long and really breaks up my workflow.

I've tried using Changes (http://changesapp.com/) for keeping things synced up, but its connection is too slow (I've never had much luck with MacFUSE). That kind of an interface, where you can see at a glance what's being updated, is what I'd love to see, though.

Probably be a major interface headache to implement, but such is my perfect world.

[Amiga2000]

it looks like everybody understands this synchronization as a tool for local vs. remote server. why not offering this e.g. for two local folders or volumes?

[edified]

rsync can do local. Don't forget server-server.

[dr_bonzo]

Another voice for TC-way.

Yes, TC's synchronize is good. So is Krusaders one.
+ [very optional] double click to see the diff of two files (if it is possible [text files], or show both images etc.)
Krusader:
http://www.krusader.org/handbook/syncdir.png
+ its wiki page http://www.krusader.org/handbook/synchronizer.html

[math]

rsync can do local. Don't forget server-server.

Yes, but only 1way sync. If you have to sync a folder with a shared volume/folder you have to use a 2way sync ( sync in both directions) tool like unison (http://www.cis.upenn.edu/~bcpierce/unison/ )

If you have to sync a project folder (with several contributors) with your notebook a bidirectional synchronization is crucial.

Maybe Changes (http://changesapp.com/) could be an good example for a dual-panel view sync tool.

[Kevin Petajan]

I have recently gone Mac and had been using Directory Opus (Directoryopus.com) on Windows for years and Amiga before that. It's folder synchronization tools are simple but very powerful. I've been looking for its Mac equivalent.

Forklift looks great, but the file/folder synch is key to my work. I hope Forklift will implement something soon!

Thanks,
Kevin

[aceman3000]

Take a look at CuteFTP Pro. It lets the user define some roles for up- and downloadloading. In example: During upload/download when filedate is newer/older/same and filesize is larger,smaller,same then overwrite/skip the file.
This is extremly handy. So you can tell CuteFTP to sync the whole directory, but it only up- and downloads the files based on your roles.

[Zaphod64]

TotalCommander (www.ghisler.com, resp. www.totalcommander.com) is THE "State of the Art" tool if you seek a tool with perfect synchronization functions between files and directories.

If you may adapt them, ForkLift will become one of the best tools for Mac ever available.

Maybe it's possible for Binarynights to contact Chrsitian Ghisler and start a productive and successful collaboration with him...

Please check also TotalCommanders file comparing functions (Files => Compare By Content), they may check ASCII/textfiles as well as unicode and binary files. And you may select/check manually between "Binary Mode", "Unicode Mode" and if the compare should be "Case Sensitive" or not.

I hope, I could give some helpful hints.

CU,
Roland

[funtuva]

Yes, I cannot agree more with Zaphod about Total Commander as the best model to follow for the Firklift sync module.

If we can get something like that, Forklift will take off as the best dual pane file manager in the Leopard market. I have been using Christian Ghisler program for many many years and have emailed back and forth with him during new releases, suggesting features, etc. If the Forklift Team could get Christian´s feedback and some leads on the development of a good sync module, it will be a sure success.

The only think similar I have seen for mac is Syntevo´s synchro program, but I prefer much more Total Commander approach, simplicity, adaptability, speed and efficiency comparing everyhting from simple files to huge dirs.

[David Hopcroft]

Hello,

I have just purchased a copy of forklift because I needed to Synchronize two directories, between my Mac and an external drive. I was able to do this with "Sync Browsing" option. This feature helped a lot but was not without issue. It was certainly a lot better than using finder :( I do tasks like this about 2 times a month.

The problems I had were:

a) If the two directories were longer than the screen size, the two file panes did not synchronously scroll together. The made it hard to see which files were missing from the the list.

b) There is little information as to the which files and directories are the same. I have to do a command+I on each pair of folders to see if the file counts match

c) When copying files from a fat32 drive files are copied across with the Locked attribute set which causes me problems in MacOS, it would be nice to ignore such attributes from FAT formated drives, all it means, to me, is the file/directory once existed on media such as CD ROM.

Anyway they were the things that caused me the most issues.

People have mentioned "Total commander", as yet no one has mentioned the windows tool "Beyond Compare" from "http://www.scootersoftware.com/". The above task would have taken about 10 Minutes with this tool. It is one of the programs I really miss from windows. It allows comparison of text files, folders, archives and ftp sites etc.. And then the merging/sync'ing of the two sources.

If I wanted a great tool to make sure to things are the same (irrespective of OS) "Beyond Compare" is the tool I would use. Plus some features like scripting tasks, would be nice.

Regards
---
David Hopcroft

[Dave61]

If I wanted a great tool to make sure to things are the same (irrespective of OS) "Beyond Compare" is the tool I would use.

I'll second that; when I need to wrangle multiple folders and files in the world of Windows, Beyond Compare is my tool of choice.

I am new Mac user, with a Mac Mini and a large external drive. Currently I am moving files from several external drives, attached to my old WinPC, and a NAS to a new larger external drive connected to my Mac. Over the years a number of duplicates were created so I have been looking for good file manager/synchroniser to help me restore order. So far I have not found a decent equivalent to BC. The thing I am really missing is the ability to filter the comparison results according to status (left/right side newer, orphans and so on).

[wajones]

You might try Diffmerge for the Mac, it works great and it's free http://sourcegear.com/diffmerge/downloads.html

[curiouslearn]

Is the latest version capable of synchronization? Total Commander for windows offers fantastic synchronization capability. I am looking for a Mac program that can do something similar. Before I try your new version, it would be helpful to know if the synchronization has been included in the latest release.

Thanks.

[csaba]

Is the latest version capable of synchronization? Total Commander for windows offers fantastic synchronization capability. I am looking for a Mac program that can do something similar. Before I try your new version, it would be helpful to know if the synchronization has been included in the latest release.

Thanks.

Hi,

No, our synch module will not be included in the 1.6 release. It is very likely going to make it for the next release.

have a nice day

[curiouslearn]

Hi,

No, our synch module will not be included in the 1.6 release. It is very likely going to make it for the next release.

have a nice day

Thanks for your reply. The quick reply that one can get on issues and problems makes your software very attractive. However, one of the main reasons I am looking for a file management software is the synchronization feature. Unfortunately, I will have to wait.

[kroxx]

Sync features on the screenshot look very promising. What I'd like to have is doing a way of 1 click syncjobs. This would be a group named "Syncjobs" in the Sidebar in which lie preconfigured syncjobs (local to S3, local to FTP etc.). These jobs will be executed by 1 click.

Again, a doubleclick in an empty part of the tabbar should generate a new tab (like in Safari). Thanks!

[rthrash]

Hi,

No, our synch module will not be included in the 1.6 release. It is very likely going to make it for the next release.

have a nice day

Any update on the status of the Sync code now that 1.6 is in the wild? This truly will make Forklift the file handling tool of choice on the Mac. Also any decision as to the general direction being taken?

[Mudi]

Any update on the status of the Sync code now that 1.6 is in the wild? This truly will make Forklift the file handling tool of choice on the Mac. Also any decision as to the general direction being taken?

You can find some info here:
http://www.binarynights.com/blog/?p=116

[funtuva]

The 116 post with the snapshot is quite old, you have any other "news" on when the sync module will me available? This is what will make FL take a jump ahead of others like DiskOrder or Mu Commander, etc...