Mirroring Repositories with svnsync

Terminology

To best discuss svnsync without getting confused, we should establish some common terminology before going any further:

  • Master: The live read/write repository that will be mirrored via svnsync.
  • Mirror: The read-only repository that is synchronized with the master via svnsync.

Overview

svnsync is a utility that became part of the standard Subversion offering when 1.4 was released and is described as a program that “provides all the functionality required for maintaining a read-only mirror of a Subversion repository.” While understanding the purpose of svnsync based on it’s documentation is simple, why would maintaining a mirror repository be important in the enterprise? With each Subversion implementation being different, there can be many reasons but there are a couple common reasons as to the importance of maintaining mirror repositories:

  • Provide a backup repository: This can be beneficial for failover, soft-upgrade, etc.
  • Provide a simple read-only repository: Some people want a simple way to provide read-only access to a repository. With svnsync, you can easily achieve this without maintaining authorization files and such. (For example: To maintain a community access point to a repository while using a different repository for the actual developer actions.)

These are just a couple examples but should give you an idea as to what value svnsync can provide. (For a more detailed explanation, please refer to the “Repository Maintenance” chapter of the “Version Control with Subversion” book.) While I could jump right in to script suggestions and examples, doing so would be a shame. To really understand why we are doing what we are, we should really understand how svnsync works. I will be brief in my explanation and then we will go into example scripts and suggestions you can apply in your Subversion implementation.

Understanding svnsync

The way svnsync works is actually pretty simple: Take revisions from one repository and “replay” them to another. This means that the mirror repository plays by the same rules as the master repository. The user account performing the actions against the mirror repository must have write access to that mirror repository. The “secret sauce” that makes svnsync work is due to Subversion maintaining the necessary metadata to know what needs to be synchronized in special revision properties on revision 0 of the mirror repository. That is it. That is how svnsync works and although it is easy to understand, to make svnsync work as designed, there are a few “rules” you need to be aware of. The following is a list of rules and/or best practices for using svnsync:

  • The synchronizing user needs read/write access to the complete mirror repository.
  • The synchronizing user needs to be able to modify certain revision properties.
  • The mirror repository needs to be read-only for all users except the synchronizing user.
  • Before you can synchronize a mirror repository with the master, the mirror repository needs to be at revision 0.

Now that we know what svnsync is and how it works and why it might be useful, let’s learn how we can start synchronizing a mirror repository with our master repository using svnsync.

Implementing svnsync

The only real prerequisite for implementing svnsync is to have a repository that you want to mirror already created prior to starting this process. Once that is complete, you can follow the steps outlined below:

Step 1: Create Mirror Repository

svnadmin create MIRROR_REPO_PATH

Step 2: Make Mirror Repository Only Writable by Synchronizing User

To make the mirror repository only writable by the synchronizing user, which in our example will be “svnsync”, we have a few options. One option is to use the authz functionality of Subversion with a default access rule like this:

[/]
* = r
svnsync = rw

The other option is to use the start-commit hook to check for the svnsync user. Here is an example, as a shell script:

#!/bin/sh

USER="$2"

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may commit new revisions as this is a read-only, mirror repository." >&2
exit 1

Step 3: Make Mirror Repository Revision Properties Modifiable by Synchronizing User

To do this, we need to create a pre-revprop-change hook with something similar to the following example, as a shell script:

#!/bin/sh

USER="$3"

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may change revision properties as this is a read-only, mirror repository."  >&2

exit 1

Step 4: Register Mirror Repository for Synchronization

Perform the following svnsync command on any system:

svnsync initialize URL_TO_MIRROR_REPO URL_TO_MASTER_REPO --username=svnsync --password=svnsyncpassword

If everything is configured properly, you should see some output like this:

Copied properties for revision 0.

Now that you have registered your mirror repository for synchronization with the master repository, we should go ahead and perform the initial synchronization so that the mirror and the master repository are synchronized.

Step 5: Perform Initial Synchronization

To make sure everything is ready and to perform the initial synchronization, on any system, perform the following:

svnsync synchronize URL_TO_MIRROR_REPO --username=svnsync --password=svnsyncpassword

If everything synchronized property, you should see some output similar to this:

Committed revision 1.
Copied properties for revision 1.
Committed revision 2.
Copied properties for revision 2.
Committed revision 3.
Copied properties for revision 3.
…

Step 6: Automate Synchronization with post-commit Hook

Now with the initial synchronization out of the way, all that needs to happen now is to write a script to be ran either as a scheduled process or as a post-commit hook to synchronize your mirror repository with the master repository. I suggest the post-commit option as it gives you the best chance of having a mirror repository as up-to-date as possible.  Here is an example hook that might be used on the master repository to synchronize a mirror repository as part of the post-commit hook.  As a shell script:

# Example for synchronizing one repository from the post-commit hook
#!/bin/sh
SVNSYNC=/usr/local/bin/svnsync
$SVNSYNC synchronize URL_TO_MIRROR_REPO --username=svnsync --password=svnsyncpassword &

exit 0

That is it. Once you have followed the steps outlined above, you should have a mirror repository that is kept up to date automatically when someone modifies the master repository. This also concludes our introduction to svnsync and how to implement it.

Tagged with: , , ,
Posted in Subversion
47 comments on “Mirroring Repositories with svnsync
  1. Malcolm Rowe says:

    One additional thing to consider: if you allow revprop changes on the master, you may also wish to create a post-revprop-change hook to re-sync the updated revprops to the mirror (via ‘svnsync copy-revprops’).

  2. Jeremy Whitlock says:

    Malcolm,
    Great idea. Thanks for contributing.
    Take care,
    Jeremy

  3. One more thing for security-concerned: the command line arguments to svnsync (including username and password) may be visible to other users in the system.

  4. Matt Doar says:

    What happens when the connection between the master and mirror is occasionally down? I presume that the next svnsync will pick up from the last successful sync, but does the non-zero exit code produce user-visible warnings in the post-commit hook?
    Handy article, thanks.

  5. You probably would not want to rely on returning a message in the hook as that would go to the programmer doing the commit. They could easily be some junior programmer that has no clue what the message means, or they could just not report it.
    If doing the synch immediately was a priority, you would want to do something like send an email or post an alert.
    But yes, the next synch will just pick up the revisions.
    Mark

  6. Thanks for the article. It would be really nice if you made some mention of rsync, and why svnsync is superior for the job. I think a lot of people default to using rsync for mirroring in general. What I’ve gathered is simply that svnsync might be more efficient since it’s transactional and understands svn’s internals. That was really hard to google, and I don’t know about the source I found: http://mail-index.netbsd.org/tech-repository/2007/07/30/0000.html
    I need to be able to convince my group that we should take the steps to set up svnsync with some justfication that it is better than our existing rsync solution (presently done with CVS).

  7. Micah,
    Thanks for the feedback. I do think you’ve found the right information. With svnsync, you have a Subversion tool doing your Subversion mirroring.
    Take care,
    Jeremy

  8. Jon Dahl says:

    Jeremy,
    First, great article. I would like to point out the hook scripts didn’t work until I made them executable. Maybe add that nugget to the article.
    I’m actually not doing a realtime push. I’m doing a pull every hour – setup with a cron job. Reason being is because the network our active server is on is not allowed to initiate a connection to our mirror network over vpn.

  9. Oliver Frank says:

    Jeremy,
    Thanks for the article!
    We are thinking about migrating to SVN. One thing that’s not to us yet is about multi-site repositories. As I understand svnsync, its purpose is only for having a redundant read-only repository. What about having a copy of a repository which also allows users to commit? This might be for performance issues over a large geographical scale.
    The right pointers or any comments would be appreciated.
    Cheers,
    Oli

  10. Oli,
    You are right. svnsync is only for creating read-only, mirrors of a master repository. Since Subversion is a centralized repository model, there isn’t anything you can do with native Subversion to make it become a multi-master setup but there is svk, http://svk.bestpractical.com/, which is a tool built on top of Subversion to allow for multi-master repositories. It might be worth a look.
    Take care,
    Jeremy

  11. Madhu says:

    Ha! Jeremy ,
    I’m actually Now doing a real time push. I’m doing a pull every hour network our active server allowed to initiate a connection to our mirror network over vpn,
    i have change Mirror network ip ,then How to sync,i what to init agine .
    are i what to sync command line but it is not working .
    pl reply
    i am waiting for you message

  12. Madhu,
    From my understanding, to “move” the slave server, all you have to do is update your synchronizing script on the master server and then update the “svn:sync-from-url” revision property on revision 0 of your master to have the new URL to the slave.
    Take care,
    Jeremy

  13. Madhu says:

    Hi Jeremy,
    How to make Automate Synchronization ,i want to write any service in windows
    can you tell me Process,can you mail me pl.
    thanks
    Madhu.

  14. Madhu,
    If you read this article, start to finish, you’ll end up with a solution that synchronizes automatically. That is, everytime your repository changes, those changes are mirrored to your slave(s).
    Take care,
    Jeremy

  15. Ryan Schmidt says:

    To answer Oliver Frank’s question from February 7, 2008, one solution would be to look into the WebDAV write-thru proxy feature which will become available in Subversion 1.5:
    http://blogs.open.collab.net/svn/2007/10/yesterday-at-th.html

  16. rajesh says:

    Hi Jeremy,
    Can you suggest anything for this scenario?
    Can there be two masters of independent “repositories”?
    Where SVN #1 is the master for Source Code #1 and a mirror for Source Code #2 and SVN #2 is the master for SC #2 and a mirror for SC #1

  17. Rajesh,
    I do not think you would get the results you seek even if Subversion did allow this. Since Subversion’s svnsync just copies revisions from one place to another, copying two separate repositories worth of changes into one single repository doesn’t make much sense. My suggestion would be to have master1 and mirror2 on the same Subversion server and master2 and mirror1 on the same Subversion server. This way, even if one box goes down, you still have valid working versions of both repositories. Since a Subversion server can server as many repositories as you deem fit, you could use this scenario with N number of repositories. Does this help?
    Take care,
    Jeremy

  18. I think the above might not be clear, but neither was the question I was answering. Basically, you cannot have a repository be a master and a slave at the same time but you can have a Subversion server host both master and mirror repositories. (The reason I mention that the question wasn’t clear was because SVN in Rajesh’s request could be a server or a repository. There was no mention of what it was so I took a stab at it.)

  19. rajesh says:

    I apologize for the late response and not being clear. Yes, I wanted a server to be able to host a master and a slave repositories and not the actual repository being a master and slave. Thanks for the response and clearing things up.
    Regards,
    Rajesh

  20. Hoang Thanh Khuyen says:

    Hi Jeremy,
    i can create the mirror of main repository but it is the read-ony repository, what will happen when my main repository down ( in some case: hdd failure….) Does everyone can temporary working on mirror repository until the main repository alive again ?And how to restore the main repository from the mirror repository.
    Thanks so much
    Khuyen

  21. Khuyen,
    Well, if your main server goes down, you would use the mirror to either recreate the master or you could make the mirror repository server your master server until the master is back up. To resync your master with the changes made to slave, you could get the last good revision on the master and dump that revision +1 from the slave and load that dump. This is standard backup procedure so please refer to the backup portion of “Version Control with Subversion”. (If you don’t already know, this book is FREE and available here: http://svnbook.red-bean.com.)
    Take care,
    Jeremy

  22. Hoang Thanh Khuyen says:

    Hi Jeremy,
    I have a problem when running command :
    [root@fedora9 ~]#svnsync initialize http://127.0.0.1/svn http://subversion/svn –username=root –password=rootpassword
    i have already set account root have read write permission !
    result :
    svnsync: DAV request failed; it’s possible that the repository’s pre-revprop-change hook either failed or is non-existent
    svnsync: At least one property change failed; repository is unchanged
    Please help!!!!!!
    Thanks so much

  23. JN says:

    I’m looking for some information on whether or not I can verify successful svnsyncs by using diff -r against the source repo against the replication repo after a successful (defined by printing no errors and returning a code of 0) svnsync. I’m having trouble determining if this can even work. No one seems to know on #svn, #svn-dev doesn’t respond at all (someone at #svn mentioned it might be a -dev kind of question), the mailing list ignores my emails, and googling using keywords like “svnsync”,”diff -r” has yielded a lot of repetitive information that would be just what I needed if I were trying to set it up and not figure out how to verify the repos match exactly.
    I’m not keen on the idea of svndumping the entire repo (twice) to diff it; I’m hoping that there is a command somewhere that will work, or that there is some simple reason why diff -r failed, because I thought that I saw it work the first two times…

  24. JN,
    Since svnsync just replays revisions from one repository to another, there is no reason why you couldn’t “svn diff -r” on master and slave to compare. I don’t know why you need to test this but I hope this helps.
    Take care,
    Jeremy

  25. David Henderson says:

    Hello Jeremy:
    I’m having some trouble with svnsync in that I get prompted for a password for every commit synced, i.e.
    Password for ‘dnadave’:
    Committed revision 46386.
    Copied properties for revision 46386.
    Password for ‘dnadave’:
    Committed revision 46387.
    Copied properties for revision 46387.
    I recently went on vacation and had to type enter for almost 400 commits. I would prefer to set up a cron job, but the password prompting prevents me from doing this. I have tried using the –username and –password flags to no avail. Is there any way to fix this problem? I’ve done plenty of google searches, but have found nothing helpful.
    Thanks!!
    Dave H

  26. Do you have your Subversion config prohibiting the caching of credentials? What version of Subversion are on your client and server? svnsync for 1.5.x changed and might be worth looking into as well.

  27. Danny O'Connor says:

    Hi Jeromy,
    I have a problem/corner-case with the post-commit hook approach with 1.4.6, it seems that it doesn’t perform well under load.
    When I set up a test with 10 users committing repeatedly I end up in this situation:
    svnsync: Revision being currently copied (2841), last merged revision (2840), and destination HEAD (2842) are inconsistent; have you committed to the destination without using svnsync?
    I presume this is happening because multiple svnsync processes are getting kicked off and getting confused with each other.
    I realize this kind of load isn’t the majority use case, but would you have any suggestions as to how to make this work ?
    Thanks a lot for the blog post!
    Danny.

  28. paul says:

    hi all,
    is it possible to synchronise two repositories which have no direct access? of course it would not be possible to do it in one step, but maybe there is an procedure. otherwise we always have to delete the mirror repository and make a hotcopy.
    cheers,
    paul

  29. Bubby says:

    Hi,

    I am using svnsync to mirror the repository but i own the files in the server as username:username. When i do a svnsync from local machine it gets updated as user root:root. What could be the cause.

  30. Scott Wells says:

    Jeremy-
    I am a newbie and am looking to do a background svnsync to another physical server.
    I am not sure what I missed, but it appears that the instructions require a URL.
    Off-forum response is also acceptable.
    TIA
    -Scott

  31. Krish says:

    Can we sync only a particular module / folder between 2 repositories instead of whole repository itself? Please add your thoughts.
    Thanks
    Krishna

  32. Bikash says:

    Hi I have got the following error while sync with mirror repo
    svnsync: Revision being currently copied (8213), last merged revision (8212), and destination HEAD (8214) are inconsistent; have you committed to the destination without using svnsync?
    I have also deleted
    #rm /your_repo_location/db/revs/8213, 8212 , 8214
    #rm /your_repo_location/db/revprops/8213, 8212 , 8214
    # in /your_repo_location/db/current back to 8211
    Again I have a problem in sync
    Thanks
    Bikash

  33. Bikash,
    It seems like you have committed directly to your mirror repository and your attempts at working around the problem, by directly modifying the repository’s directory structure probably did more damage. When using svnsync to mirror a repository, the mirror is not to be updated by anything other than svnsync. The only good way to fix this is to dump the mirror at revision 8212, or whatever revision was last successfully synced, and then resync the mirror.

  34. Thulasi dass says:

    Hi,
    I am doing the Svn replication from Master server to destination server. I have done replication all respository except one. One is hug near 27GB. While replicating I got EOL error.So I have taken the dump from master server and restore the destination server.
    Now I got the diffrrent error merged revision 0. I have refered some sites andrectified rev 0 issue. But still could not able to sync. Below error i am getting.
    svnsync: Destination HEAD (7207) is not the last merged revision (7208); have you committed to the destination without using svnsync

  35. Thulassi,
    Well, it appears you’ve got a slave server that thinks that revision 7207 was the last revision synched from the master but the slave is actually at revision 7208 which means that svnsync’s metadata is out of sync. To fix it, figure out what revision the slave really is at from the dump and update the metadata stored on revision 0 as mentioned here:
    http://whynotwiki.com/Svnsync
    Take care,
    Jeremy

  36. doug says:

    Is there a way to use sync with a cert or key, instead of plaintext password

  37. Doug,
    What would be best in this situation would be to perform some sort of Subversion command as the user svnsync is running as, so you can create a cached password and accept any ssl certificates. Then you can run svnsync without the password.
    Take care,
    Jeremy

  38. Johnny Clark says:

    With regards to Danny O’Conner’s post on 9/3/08:
    This has actually happened to my repo 2 times in the last 2 years. It effectively corrupts the mirror such that you need to recreate it. We are using 1.4, so I’m hoping that it has been resolved in later versions. So it’s not quite as unlikely as you would think.
    Great article, I was actually trying to determine whether or not the master is “locked” during the initial synch process, b/c our repo is very large and it is going to take several hours. So while the article didn’t help me today, it would have been very valuable the first time I set up the mirror a few years ago.

  39. Nehru Saini says:

    I want to setup Active Sync between Master-Master.
    Please suggest me how to do it.
    Regards,
    ~Nehru

  40. Tracy says:

    I receive an svnsync:Couldn’t get lock on destination repos after 10 attempts when I use the following in my post-commit:
    #!/bin/sh
    REPOS=$1
    REV=$2
    SVNSYNC=/usr/local/bin/svnsync
    $SVNSYNC synchronize file://${REPOS} –username=svnsync –password=svnsyncpassword
    Also not sure why you need to specifiy the full path of svnsync. Works the same without the path.
    BTW, I have done the standard svn propdel svn:sync-lock –revprop -r 0 file:///export/svn/repos/ in order to delete lock
    Regards,
    Tracy

  41. chetan says:

    Out of 5 i want to sync only 3 revisions to live server. Please suggest command for the same.

    Thanks in advance.

  42. Arman says:

    Hi Mr. Jeremy,

    hope you are doing well.

    I want to sync SVN repositories placed geographically in different places using site site VPN connection. before doing this i have created two test servers with Collabnet subversion edge installed in my lab, but i am getting some errors when i do Step 4: Register Mirror Repository for Synchronization using the command svnsync initialize URL_TO_MIRROR_REPO URL_TO_MASTER_REPO –username=svnsync –password=svnsyncpassword. I am new to SVN and i think i am doing something wrong with the pre-revprop-change hook and the command. please see my setup given below,

    Server1 (Target/ Mirror) server

    OS: Windows7
    IP: 10.150.100.158
    Repository Name: Mirror
    Repository Path: http://10.150.100.158/svn/MIRROR

    First i created a user svnsync in this svn and added the below formula to Make Mirror Repository Only Writable by Synchronizing User as mentioned in step2.
    [/]
    * = r
    svnsync = rw

    then i created a new pre-revprop-change hook by replacing the existing contents with the the one mentioned in the Step 3: Make Mirror Repository Revision Properties Modifiable by Synchronizing User. then i renamed it to re-revprop-change.bat in the hook directory of the mirror repository.

    Server2 (Source/ Master) Server

    OS: Windows7
    IP: 10.150.100.202
    Repository name: master
    Repository path:http://10.150.100.202/svn/master

    no other changes i made on this server.

    then i execute the command using command prompt in the target/mirror server,

    svnsync initialize http://10.150.100.158/svn/MIRROR http://10.150.100.202/svn/master –username=svnsync –password=svnsyncpassword

    but i receive following errors,

    svnsync: E175002: DAV request failed; it’s possible that the repository’s pre-re
    vprop-change hook either failed or is non-existent
    svnsync: E175002: Revprop change blocked by pre-revprop-change hook (exit code 2
    55) with output:
    ‘#!’ is not recognized as an internal or external command,
    operable program or batch file.
    ‘USER’ is not recognized as an internal or external command,
    operable program or batch file.
    “$USER” was unexpected at this time.

    the error message changes when i make some changes with the hook script or the syntax. Also i noticed in some steps the user mentioned is svnsyn and but in some other steps its syncuser. I tried all but no luck. it will be a great help for me if you can throw some light on this. any help is highly appreciated.

    Thanks & Regards,
    Arman

  43. We had configured SVN primary and SVNsync server. setup was working fine.
    Last month we have migrated our all servers IP after this migration we are unable to sync the main and mirror SVN server. please help me how to change the main svn server IP in mirror server repository for sync
    error ” Can’t connect to host………”

    SVN ver: “CollabNetSubversion-server-1.6.3-3.win32” windows

  44. magesh says:

    we’re getting below error while svn sync

    svnsync: Revision being currently copied (12749), last merged revision (12750), and destination HEAD (12750) are inconsistent; have you committed to the destination without using svnsync?

    Please let us know how to fix above issue.

    Thanks for your support in advance.

  45. Rafael Rojas says:

    Hi Guys.

    I a had problem with post-commit script,but was SELinux on Centos. When I disable it the script runs perfectly. Thanks for sharing your knowledge

  46. Khanh TRAN says:

    Hi Jeremy,

    I’ve encountered an error message ‘Only the “syncuser” user may update this repository.’ when I try to commit.

    How to create synuser in my repository? (I create list of users in the svn_auth_file)

    Thank for answer.
    Khanh

  47. Khanh TRAN says:

    I install CollabNet Subversion Edge 4.0.1. I don’t know how to configure the Mirror Repository to Sync, commit code to SVN server.

    Could you please tell me what I can do?

    Thank you very much
    KhanhRoy

1 Pings/Trackbacks for "Mirroring Repositories with svnsync"

Leave a Reply

Your email address will not be published. Required fields are marked *

*