Automated git deployments from Bitbucket
Important: Bitbucket have changed how webhooks function, and the technique described in this post will no longer work without modification. One of my readers has created an updated version, and I recommend trying that instead. I no longer use this deployment method and won’t be updating my tutorial or answering comments, but I have left the comment section open so that readers can post their tips and help each other out.
Git may not have been designed as a deployment tool, but for small projects it can do the job quite nicely. What makes Git deployments attractive is how frictionless the process is: make some changes to your project, merge them into your production branch, push the commit to a remote repository and like magic the changes are live! Git knows which files need to be changed or deleted, so you don’t have to think about it. If you’re already using git to version control your project then you probably won’t even need to modify your existing workflow, once the initial setup is done.
I use Bitbucket for hosting my private repositories, and have recently implemented a deployment process that integrates with Bitbucket’s POST hooks feature. There are basically three things you need to do to make this work:
- Set up SSH keys so your server can talk to Bitbucket
- Clone your Bitbucket repository on your web server
- Setup a hook on Bitbucket and an associated deployment script on your server
Here’s what your deployment workflow will look like once we’re done:
- Develop your website locally
- When you’re ready to deploy, commit your changes and push them to Bitbucket
- When Bitbucket receives the commit it will notify a deployment script on your server
- The deployment script will fetch the changes into a cloned repository on your server, and checkout files to your public web directory
Prerequisites
Before we get started, check that your server meets the following requirements:
Most shared web hosting accounts will fail at least one of those requirements, but if you’ve got a VPS or dedicated server then you should be good to go.
For the purposes of this tutorial I’m going to assume that you have a git repo already set up on Bitbucket, and that the repository’s directory structure mirrors your production website. For instance if you want an index.html file deployed to the root level of your website’s public directory, that same file will exist in the repo’s root directory.
I’m also going to assume that you will be deploying from a branch named production. In practice you can deploy from any branch other than master, but it’s a good idea to deploy from a branch that is not used for active development, so that you have control over when a deployment occurs. In my workflow I develop in the master branch, then merge master into production when I’m ready to deploy. Before we get started, make sure you’ve made an initial commit to your repository’s production branch, and pushed to Bitbucket.
I’ve tested this process on Centos, but you might need to change directory paths to suit your own server environment. Whenever you see a variable inside angled brackets in my in my code samples, such as <repo-name> or <username>, that’s a placeholder that you will need to replace with a value specific to your own project.
Now that the preliminaries are out of the way, let’s get started.
Set up SSH keys
For your server to connect securely to Bitbucket without a password prompt, it needs to use an SSH key.
On your server navigate to the ~/.ssh directory of the user that PHP runs under. I’m running the Apache suPHP module on my server, so PHP runs as the user that owns the website. On your server the web user might be the apache, nobody or www-data user. You will need to create the user’s .ssh directory if it doesn’t exist. At a shell prompt type:
cd ~/.ssh
ssh-keygen -t rsaWhen prompted either accept the default key name (id_rsa) or give your key a unique name – I chose bitbucket_rsa. Press enter when asked for a passphrase, which will generate a passwordless key. Usually this isn’t recommended, but we need our script to be able to connect to Bitbucket without a passphrase.
A public and private key pair will be generated. Copy your public key – the one with a .pub extension – to the clipboard. On the Bitbucket website navigate to Account > SSH Keys, and choose to add a new key. Paste in your public key and save it.
Back on your server, edit your ~/.ssh/config file to add bitbucket.org as a host. This ensures that the correct key is used when connecting by SSH to bitbucket.org. You’ll need to create the config file if it doesn’t exist:
Host bitbucket.org
IdentityFile ~/.ssh/bitbucket_rsaWhenever you do a git fetch Bitbucket will verify your identity automatically, without prompting you for a password.
Cloning your repository
Now that SSH is configured you can clone your Bitbucket repository on your server. You might be tempted to clone the repository directly into the public website directory (for the sake of brevity we’ll call it www), but that approach comes with significant security risks. It requires that there is a .git folder inside of www, from which a malicious attacker could extract your entire website source code, possibly including database credentials and other sensitive information. Sure, you could use an .htaccess directive to hide the .git directory, but if that .htaccess file were accidentally deleted or edited you’d be left wide open.
A better approach is to store your git repository outside the public website directory, where it is hidden from prying eyes. If you create a bare repository then you can still checkout files to a detached working tree in www.
Before we begin, navigate to your repository on the Bitbucket website and copy its SSH URL. This will be in the format git@bitbucket.org:<username>/<repo-name>.git
Navigate the location on your server where you want to clone the repository. A good spot is probably one level above the www directory, which on my system is the website user’s home directory. At a shell prompt, clone your Bitbucket repository:
cd ~
git clone --mirror git@bitbucket.org:<username>/<repo-name>.gitNotice the --mirror flag? As its name implies this flag creates an exact mirror of the source repository, including mapping it’s remote branches. It implies --bare, which means that our repository will not have a working copy.
Your repository will be cloned into a directory called <repo-name>.git, in the user’s home directory. It doesn’t really matter what you name the directory, but suffixing the directory name with .git implies a bare repo, so it’s a useful naming convention to follow.
Now let’s do an initial checkout:
cd ~/<repo-name>.git
GIT_WORK_TREE=/home/<username>/www git checkout -f productionIf this is first time you’ve communicated with bitbucket.org over SSH you may be prompted to accept Bitbucket’s server fingerprint, but if SSH is correctly configured you won’t be asked for your Bitbucket password or a key passphrase.
We have specified a GIT_WORK_TREE that corresponds to your public web directory, and checked out the production branch to that location. This step is important so that in future when our deployment script does a checkout we’re already on the correct branch.
Check that your initial checkout completed as expected, and that files from your production branch have been created in your public web directory. If everything worked as expected then you’re ready to set up automated deployments.
Create a Bitbucket POST hook
In your www directory make a new directory named deploy, containing three files: index.html, bitbucket-hook.php and deploy.log.
Add the following to bitbucket-deploy.php, changing $repo_dir, $web_root_dir and $git_bin_path to suite your server environment:
<?php
$repo_dir = '/home/<username>/<repo-name>.git';
$web_root_dir = '/home/<username>/www';
// Full path to git binary is required if git is not in your PHP user's path. Otherwise just use 'git'.
$git_bin_path = 'git';
$update = false;
// Parse data from Bitbucket hook payload
$payload = json_decode($_POST['payload']);
if (empty($payload->commits)){
// When merging and pushing to bitbucket, the commits array will be empty.
// In this case there is no way to know what branch was pushed to, so we will do an update.
$update = true;
} else {
foreach ($payload->commits as $commit) {
$branch = $commit->branch;
if ($branch === 'production' || isset($commit->branches) && in_array('production', $commit->branches)) {
$update = true;
break;
}
}
}
if ($update) {
// Do a git checkout to the web root
exec('cd ' . $repo_dir . ' && ' . $git_bin_path . ' fetch');
exec('cd ' . $repo_dir . ' && GIT_WORK_TREE=' . $web_root_dir . ' ' . $git_bin_path . ' checkout -f');
// Log the deployment
$commit_hash = shell_exec('cd ' . $repo_dir . ' && ' . $git_bin_path . ' rev-parse --short HEAD');
file_put_contents('deploy.log', date('m/d/Y h:i:s a') . " Deployed branch: " . $branch . " Commit: " . $commit_hash . "\n", FILE_APPEND);
}
?>
This script iterates over the payload object sent by Bitbucket, looking for commits made to the production branch. If any are found, a git fetch and checkout are performed and the deployment details are logged.
For security through obscurity you might choose to give your deployment script a difficult to guess name – bitbucket-hook-a13jsur5kcidwe89z.php, for example. The index.php file you created early is also a simple security measure: it stops anyone from viewing the directory index.
On the Bitbucket website navigate to your repository’s Administration > Hooks screen and add a new POST hook, pointed at http:/<domain>/deploy/bitbucket-hook.php.
Deploying
Whenever you are ready to deploy to your web server, merge your development branch into your production branch, and push the production branch to Bitbucket. Your custom POST hook will be triggered, and your deployment script will fetch the repository to the server and checkout the production branch to your web root.
Hey presto! With one commit your changes have been automatically deployed to your production web server.
My instructions might look fairly complicated, but after you’ve followed the steps once or twice it actually becomes really fast to set up.
Troubleshooting
Here are a few things to check if deployments aren’t working as expected.
File permissions
You should make sure that the web user (the user that PHP runs as) has permission to write to your local git repository. The easiest way to ensure this is for that user to own the repo:
chown -R <user>:<group> <repo-name>.gitMake sure that the same user also owns its ~/.ssh directory and contents.
Git path
You may find that you are unable to perform git commands using exec since the git binary is not in your PHP user’s PATH. This can be solved by including the full path to the git binary in your deployment script, for example:
$git_bin_path = '/usr/local/bin/git';To find where your git binary is located, run this shell command:
which gitHat tip to Jonathan Johnson for this one. His article might help solve other issues you’re having, too.
Payload
If you need to examine the Bitbucket payload that’s being sent to your deployment script, add the following line to the top of bitbucket-hook.php:
file_put_contents('deploy.log', serialize($_POST['payload']), FILE_APPEND);87 thoughts on “Automated git deployments from Bitbucket”
Comments are closed.
Seems there is a merge step missing (unless I am not understanding somehting).
The fetch command will return
* branch HEAD -> FETCH_HEAD
Then I seem to need to do a merge command like so
GIT_WORK_TREE=/home/<username>/public_html git merge FETCH_HEAD
If I do the “checkout” command it doesnt seem to update to the new branch at all.
Does that sound correct?
Hi Marcus. The instructions as they appear in the article work for me – no merging required. From what I understand merging could be problematic since you would have no opportunity to resolve conflicts, if for instance any files had been changed locally on the server. That’s why I do a checkout with the -f flag, since it throws away any local changes.
However I don’t pretend to be a git expert, so you might find that a merge works better for you!
One step that is essential for my process to work is the initial checkout, which sets the detached work tree to the correct branch:
GIT_WORK_TREE=/home/<username>/www git checkout -f production
If you did that step, and it worked successfully, then I think subsequent checkouts (i.e. from the deployment script) should work as expected. You could confirm this from the command line by issuing the same git commands as the deployment script:
git fetch
GIT_WORK_TREE=/home/<username>/www git checkout -f
More of a GIT expert than me it seems. The one subtle difference I made was initially cloning the repo on the live server using –bare instead of –mirror. After re-cloning using –mirror it does indeed work as described. No merging needed.
I had miss-understood the difference between –bare and –mirror and thought that –bare would be the same but require less storage.
Thanks very much. I am perhaps making the post hook PHP script a little more configurable but I love that you’ve provided the minimum code for simplicity.
@Marcus I ran into the exact same thing actually! I had also assumed that –bare would work, but found that –mirror was required to replicate all the branches.
Hi Jonathan,
Great tutorial! Appreciate your explanation in simple steps.
Although, I am facing few issues while updating my branch.
Few things to note:
1. My ‘production’ branch is ‘master’.
2. I was recently provided with Admin privilege for the repository. Earlier I was cloning it using username@bitbucket.org format. Now git@bitbucket works
I am using Amazon EC2 Ubuntu where my username is sampleusername, and it has right to read/write on ~ and www directory.
My problem is that whenever I try to use the hook, or try to fetch manually. Following error displays:
fatal: Refusing to fetch into current branch refs/heads/master of non-bare repository
fatal: The remote end hung up unexpectedly
I believe –mirrror repos are also master. Then why does it fails to recognize it.
It would be great if you could help me. Thank you.
The tutorial was really helpful, thanks. Came across a couple of issues:
Something may have changed since this was written, but in my case I found that the payload JSON coming from Bitbucket is string escaped. The documentation on the Atlassian site seems to indicate it shouldn’t be, but it definitely was for me.
It worked when I added stripslashes to line 9 in the hook script:
$payload = json_decode(stripslashes($_POST[‘payload’]));
Another slight hiccup:
If you’re creating the SSH config file because it doesn’t already exist, you need to change the permissions on it with a “chmod 600 config”, or the SSH access will fail.
(Of course, as soon as I got it all working I discovered at the very end my host doesn’t support exec …)
Sorry, can’t edit my comment. Forgot to add there’s a typo when you’re entering your POST hook details on the Bitbucket site, you should specifiy the URL as:
http://yourwebsitename.com/deploy/bitbucket-hook.php,
Instead of just:
http://deploy/bitbucket-hook.php.
@Mick Thanks for sharing your tips. I haven’t yet had to run the bitbucket payload through stripslashes, but I’ll keep an eye out for that one! I’m sorry to hear that it wasn’t until the end of the process that your discovered you can’t enable exec in your environment :(
PS. Thanks for spotting the typo. I had forgotten to escape my angled brackets, so <domain> was rendering as an HTML tag.
I have amended the deployment hook PHP script to account for two scenarios that would previously have caused the Git fetch & checkout to fail silently:
1) When doing a merge and push to bitbucket the payload’s
commitsarray is empty, making it impossible to know which branch was pushed to. I’ve chosen to err on the side of caution and do a fetch & checkout in these cases – it’s better to occasionally do a checkout when it’s not required than to risk not doing one when it is needed.You can read more about this in Bitbucket issue 1427.
I’m not sure how I overlooked this quirk, since merge and push is the deployment workflow I recommend in the article. Sorry for any confusion!
2) When pushing multiple commits to Bitbucket simultaneously, the payload’s
branchvariable is null. Instead, details about which branches were committed to are contained in an array namedbranches. The script now checks that array for the string “production”.The repository that I am delaying contains a few submodules. When checking out my repository into my GIT_WORK_TREE, none of my submodules are being checked out; presumably because the parent repo is mirrored and bare.
How would I handle submodules that are referenced in my repository?
Excellent tutorial, thank you.
@Brian I don’t have any experience working with submodules, but it sounds like the –recursive option might do what you want: http://stackoverflow.com/questions/3796927/how-to-git-clone-including-submodules
Hi Jonathan,
I also have similar setup as mentioned by Utkarsh and i am facing the same issue while fetching the data on server. Can you please help me out?
Thanks.
@Utkarsh @Ankur I don’t think my method will work if your deployment branch is ‘master’. In my tests I wasn’t able to checkout the master branch after cloning the repository, since I was already on the master branch.
I have updated the article to be clear that you should deploy from a branch other than master.
If either of you do have success deploying from master let me know how you went about it.
For me, the bitbucket-deploy.php script will not work. I think the problem is that the apache user (daemon) does not have sufficient permissions, because when I turn on error reporting it says:
Warning: file_put_contents(deploy.log) [function.file-put-contents]: failed to open stream: Permission denied
This daemon user does not have a home directory. Instead, I did the ‘setup ssh keys’ step in the home directory of both the ‘root’ and ‘www’ users but clearly these users have nothing to do with daemon.
How can I go about giving daemon user the right permissions and access to the ssh key?
Update:
I have now run these commands:
chown -R daemon ~/repo-name.git
chown -R daemon /var/ebs/www/
Now, the bitbucket-deploy.php runs without any error. However, the files are not copied to the public www directory. Each time I run the php file deploy.log is updated with a new line like this:
N;03/08/2014 03:54:45 am Deployed branch: Commit:
@ZC You say “Each time I run the php file”. Do you mean that you’re running the file manually (i.e. by visiting its URL in your web browser), or is the file being executed by your Bitbucket POST hook (i.e. automatically after you push to your Bitbucket repo)?
If you’re trying to execute the PHP file manually then I would expect it to fail. However, it _is_ possible to create a deployment script that can be run manually, something like this:
Assuming everything else is correctly configured, executing that PHP file will do a git fetch and checkout from your repo. This is a good way to test that everything is correctly configured on your server.
If you can successfully do a manual deployment by running the script above, then you can be fairly sure that your automatic deployment is failing because of a problem with the Bitbucket POST hook. I would start by examining the payload that is being sent from Bitbucket. I cover this in the troubleshooting section of the article.
Thanks for this post! Very helpful.
Is it possible to alter the post hook so that it will work with multiple branches that have different work trees?
For example: I commit local development changes to the master branch, and push to bitbucket, then have a staging branch that when merged (from master) and pushed to bitbucket would deploy to staging.example.com, and a production branch that would do the same, but deploy to example.com (staging and production on the same server).
I image it would include something like defining
$staging_root_dir = ”
$production_root_dir = ”
Thanks,
Eric
When I execute that php file, it gives no response and deploy.log is not updated.
I think exec is working on the server, because if I make a simple php file with only this inside, it gives the output ‘exec works’:
if(exec(‘echo EXEC’) == ‘EXEC’){
echo ‘exec works’;
}
I wonder if there is a syntax error in the php you gave above? I tried adding these lines to show errors but it always results in a blank html file returned:
error_reporting(E_ALL);
ini_set(‘display_errors’,1);
echo exec(‘whoami’);
If I remove your code, so only the three lines above remain, then it outputs ‘daemon’ when executed in the browser.
By the way I changed the apostrophe character in your code so that ‘ is now ‘
Please ignore the comment above.
I have executed your code, and found that it adds a line to deploy.log as follows:
03/13/2014 01:26:52 am Commit:
But it doesn’t seem to have done anything as the files in web root have not been updated.
The full php is like this:
error_reporting(E_ALL);
ini_set(‘display_errors’,1);
echo ‘start’;
$repo_dir = ‘/root/tvvtest.git’;
$web_root_dir = ‘/var/ebs/tvv’;
// Full path to git binary is required if git is not in your PHP user’s path. Otherwise just use ‘git’.
$git_bin_path = ‘/usr/bin/git’;
// Do a git checkout to the web root
exec(‘cd ‘ . $repo_dir . ‘ && ‘ . $git_bin_path . ‘ fetch’);
exec(‘cd ‘ . $repo_dir . ‘ && GIT_WORK_TREE=’ . $web_root_dir . ‘ ‘ . $git_bin_path . ‘ checkout -f’);
// Log the deployment
$commit_hash = shell_exec(‘cd ‘ . $repo_dir . ‘ && ‘ . $git_bin_path . ‘ rev-parse –short HEAD’);
file_put_contents(‘deploy.log’, date(‘m/d/Y h:i:s a’) . ” Commit: ” . $commit_hash . “\n”, FILE_APPEND);
The output is only ‘start’ in the browser.
There is a ‘production’ and ‘master’ branch on bitbucket.
I have tried to debug this by using command prompt.
As ‘root’ user I run the following command:
cd /root/tvvtest.git && git fetch
Result:
Bad owner or permissions on /root/.ssh/config
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
It also doesn’t work when the php file is launched in the browser.
All directories/files in /root/.ssh are owned by daemon user and daemon group (of which root user is a member). Permissions are 755 (-rwxr-xr-x).
By the way I found that the HOME directory of daemon user is /sbin/ so I made a .ssh directory there and copied the files from /root/.ssh to it and made daemon:daemon the owner. But it seems that /root/.ssh is used, not /sbin/.ssh, as shown in the error above.
@ZC If it helps debug your SSH connection issue, here are the permissions on the .ssh directory of one of my sites:
drwxr-xr-x 2 myuser myuser 4096 Dec 21 22:37 ./
drwx–x–x 15 myuser myuser 4096 Mar 9 07:36 ../
-rw——- 1 myuser myuser 417 Dec 21 21:27 authorized_keys
-rw——- 1 myuser myuser 1675 Dec 21 21:26 bitbucket_rsa
-rw-r–r– 1 myuser myuser 408 Dec 21 21:26 bitbucket_rsa.pub
-rw-r–r– 1 myuser myuser 67 Jan 12 20:01 config
-rw-r–r– 1 myuser myuser 806 Dec 21 22:38 known_hosts
I am still trying to get this to work without success. Forget my earlier posts, the situation now is this…
Pushing a change to ‘production’ branch on bitbucket triggers the POST hook to my server. Results are as follows:
1) deploy.log is updated as follows:
03/18/2014 01:48:16 am Deployed branch: production Commit: 4bc033a
However, this is not the latest commit on that branch. This is the original commit when I first made the production branch.
I am outputting the payload to deploy.log, and this payload refers to the latest commit or “node” (d12e472d98f5) not the original commit (4bc033a).
2) the $repo_dir .git directory is updated
3) the files in $web_root_dir remain unchanged (probably because they match the original commit)
So, as you can see it is *nearly* working but it just will not fetch the latest commit for some reason.
Some further info:
4) If I run the commands from command prompt, it works as expected (the latest commit on production branch is copied to web root)
cd /var/ebs/tlv2/tvvtest.git && /usr/bin/git fetch
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From bitbucket.org:user/tvvtest
c4bc360..4763637 production -> production
cd /var/ebs/tvv2/tvvtest.git && GIT_WORK_TREE=/var/ebs/tlv checkout -f
5) Next, on bitbucket I edit a file on production branch. This triggers the post hook. The payload data mentions the latest commit (ce7…), however the previous commit (476…) from when I ran it from command prompt in (4) above instead is deployed.
So to repeat, for some reason when the commands are run from command prompt it works but when run via post hook + deploy.php it won’t deploy the latest commit.
@ZC, Did you originally clone with –mirror or –bare?
I had all sorts of issues with –bare, but –mirror worked fine.
Other than that, it sounds like a permission issue of sorts, if it works via command line but not from the PHP user, perhaps the PHP user has write access to the git directory but not the web root. Really hard to say, can you get any more debug info to see what git is actually doing?
I used –mirror as per the instructions above.
It might be a permissions problem, however I am not sure how to solve it.
The php user is ‘daemon’ so I have set the owner:group to daemon:web_creator for everything in both the web root and the .git folder. I also allowed read/write permissions for that group:
$ chgrp -R web_creator .
$ chmod -R g+rw .
The deploy.php happily updates files in the .git directory, but not web root. I think this is because it is deploying the outdated commit and therefore there is no need to update web root which already has the files from that commit.
When I run this command from command prompt, it also returns the value of the old commit, not the latest commit:
git rev-parse –short HEAD
@ZC Sounds like a permissions issue, probably on the .git directory.
I ran into a problem very similar to yours after performing a fetch on my repo while logged in as another user (i.e. not the user PHP runs under). Git wrote a bunch of files to the repository’s objects directory, and these new files were owned by the wrong user. Subsequent fetch attempts by the PHP user failed because that user didn’t have permissions to edit the newly created sub-directories. The lesson I learned is that the only user who should be doing
git fetchis the Apache user – daemon in your case – otherwise permissions can get mucked up.It might be a silly suggestion, but looking at your last comment is seems like you’re recursively setting the group of the repository, but not its owner? Maybe try:
$ chown -R daemon:web_creator .An interesting issue I’ve come across.
When running the fetch command withing mirror.git I get this now.
———————-
From bitbucket.org:user/reponame
! [new branch] dev -> dev (unable to update local ref)
a50dfe2..0c7346d master -> master
error: some local refs could not be updated; try running
‘git remote prune origin’ to remove any old, conflicting branches
—————————
I think this is due to us removing a branch entirely from the repo. I’m wondering if a prune command should always be run first before running the fetch or something. For auto deployment, we probably don’t care if we force prune all the time?
running this
git remote prune origin
before the fetch command made everything kick back into action again.
I’m having some issues trying to get SSH authorization working. I did every step as you say in your post but all the time when I run the command cd /home/mapyet/mapyet2.git/ && git fetch I need to supply my bitbucket password. This are the permissions of files:
drwxr-xr-x 2 root root 4096 Apr 26 16:26 authorized_keys
-rw——- 1 root root 1675 May 16 11:18 bitbucket_rsa
-rw-r–r– 1 root root 406 May 16 11:18 bitbucket_rsa.pub
-rw-r–r– 1 root root 53 May 16 11:21 config
And this is the content of config file
Host bitbucket.org
IdentityFile ~/.ssh/bitbucket_rsa
What I miss? Also how I do know if Bitbucket is making a POST in each PUSH I made to the main reposotory?
Thanks for the tutorial. I have most of this up and running. The initial setup works and grabs the repo and files but when push more commits to the repo the exec commands don’t seem to fire. I am getting info in the log but the files are not updating.
Could this be a permissions issue?
I’m having some issues getting this set up.
If I type the commands manually into the terminal, it works fine, but when I try to push to my production branch, I get a php error:
Undefined variable: branch in bitbucket-hook.php line 34
and also an error:
sh: 1: cd: can’t cd to my $repo_dir
I also get the cd error when running the deploy.php example you added in the comments above. It seems like a permission error, but PHP is running as the same user that owns all of the files, so I’m not sure what’s going on.
I fixed the cd error — I was using shortcuts, but when I changed it to the full path, it worked fine. I’m still getting the undefined variable error, though.
So I have been using this article religously as well as the deploy script. The only issue I am having is this.
Let’s say I have a new wordpress site remotely. The wordpress install functions fine. As soon as I run the following command:
GIT_WORK_TREE=/home/usernm45/public_html git checkout -f master
My public_html file permissions get screwed up, and I get 500 INTERNAL SERVER error. I have a script to reset all of the WordPress permissions back to how they should be, but on regular sites, I can’t run the script, because folder names may be different than wordpress.
Any suggestions?
That is the only hiccup I have. Other than that, this is a great article!
@ZC: I experienced the same issue, I restart apache server and it worked
A bit of modification to debug if apache failed to execute git command, add 2>&1 to route stderr to stdout
$output = “”;
exec(‘cd ‘.$repo_dir.’ && ‘.$git_bin_path.’ fetch 2>&1′, $output);
exec(‘cd ‘.$repo_dir.’ && GIT_WORK_TREE=’.$web_root_dir.’ ‘.$git_bin_path.’ checkout -f 2>&1′, $output);
file_put_contents(‘deploy.log’, $output, FILE_APPEND | LOCK_EX);
$commit_hash = shell_exec(‘cd ‘.$repo_dir.’ && ‘.$git_bin_path.’ rev-parse –short HEAD 2>&1′);
big thx for the great tutorial!
everything works except git fetch command.
if i go to the git repository and try git fetch there it works.
if i do it with the php script it doesn`t.
do you have an idea why git fetch does not work with the php script?
git checkout command works with the php script …
Thanks for the great tutorial!
i`ve a question regarding the ssh part.
everything works fine as long as i use ssh agent and do an ssh add of the private key.
but as soon as ssh agent stops it`s not working any more and i get a:
Host key verification failed.fatal: The remote end hung up unexpectedly
error.
what can i do making it work without ssh agent?
thank you!
Thanks for the guide I just got it working for my site after dealing with permissions issues for the deploy folder and a typo in the web_root_dir .
If you have a repository on the development web server, chances are the files that are being uploaded directly the server are for quick tests (instead of having a bunch of commits for testing)… with that being said, it’s usually OK (based on your internal process) to use “git fetch origin” followed with “git reset –hard origin/master”.. but there are caveats, you just need to know which process to do.
Thanks very helpful…!
One aclaration: The user is always the bitbucket user. If you are getting a different user from the certificates (I think that you can know that from the file .pub at the end) you will get an error.
To fix that you need to user ssh-keygen -C “e@mail.com” and the magic appears!
This is an excellent article and one that I’ve found very useful. However, I have some problems with the permissions side of things. All other similar PHP Git deployment scripts I’ve seen on the web also seem to skate over this issue.
Firstly, the approach of having a PHP script triggered by an HTTP request means that both the DETACHED HEAD repo and the deployed files must all be owned or at least writeable by the web server user (i.e. apache or www-data etc). This is really not very good practice.
Secondly, I just don’t like the idea of having the www-data user (which is a user account that can access all the different system user accounts’ website files etc) having an .ssh key that gives it access to an entire Bitbucket account which is only related to one of the said websites for example.
This latter issue has been resolved by Bitbucket in that they now offer “Deployment Keys” which is a per-repository setting for submitting public ssh keys that provide read-only access to only the specified repository. This is a vast improvement on using a global public key that gives read/write access across all repositories in the Bitbucket account.
I have taken Jonathan’s deploy PHP script and refactored it. Essentially it’s now in two parts where the Bitbucket Hook POST request simply triggers the creation of a data file. The script is also set up as a cronjob to run every minute or whatever. The cron’d executed copy of the script simply looks for the data file, and if it exists it executes the git checkout etc. This means that the local repository and the deployed web site files can all simply and easily be owned by the actual system account rather than by www-data.
I have packaged it up over on GitHub – https://github.com/benr77/BitbucketCronDeploy
Any feedback or contributions more than welcome.
Thanks
I think there’s a typo error, you said in first paragraph “bitbucket-hook.php”
In your www directory make a new directory named deploy, containing three files: index.html, bitbucket-hook.php and deploy.log.
But in the second paragraph you said that we must change in bitbucket-deploy.php
Add the following to bitbucket-deploy.php, changing $repo_dir, $web_root_dir and $git_bin_path to suite your server environment:
It’s bitbucket-deploy.php or bitbucket-hook.php ¿?¿?¿?¿?¿?
Thanks for the tutorial. I found it very useful. :D
GIT_WORK_TREE=/home/path/www
This critical step doesnt seem to work on all git versions..
A definitive way of setting this would be:
git config core.worktree = /home/path/www
basically does the same thing (modifying the git config file) but actually works with all git versions
Unable to use this script on a Windows host with Apache and PHP configured.
When i try to execute the git fetch and any other command that requires the ssh to connect, the browser just locks up. Anyone else have any idea?
What can be used instead? HTTPS? Is that safe?
I had the same issue with submodules as reported by Brian – they were not being checked out nor updated in the process as described in the article. So I took the basic ideas from this article and went down a different route, where instead if “git fetch” I run:
git pull –recurse-submodules=yes
followed by:
git submodule update
which works on a regular repository, with a working tree included, but located outside of my web-accessible directories. Then I use rsync to copy the files to the public web directory:
rsync -a –exclude=”.git*” /home//git/my_repo/ /home/public_html/
Fantastic article, but you forgot to mention that you have to add the ssh key by using:
ssh-add ~/.ssh/bitbucket_rsa &>/dev/null
Jonathan, Thank you for taking the time to post this solution.
I found it extremely helpful when crafting a solution for my Coldfusion server.
I converted your php code to coldfusion. Please feel free to use it on your blog or share it as you see fit.
http://pastebin.com/tVVK0c7N
All the best,
Steven Benjamin
First of all thanks for tutorial!
My SSH settins:
-rw-r–r– 1 web-user web-user 381 Jul 10 12:38 authorized_keys
-rw——- 1 web-user web-user 1679 Dec 1 16:42 bitbucket_rsa
-rw-r–r– 1 web-user web-user 409 Dec 1 16:42 bitbucket_rsa.pub
-rw——- 1 web-user web-user 78 Dec 1 16:42 config (-rw-r–r– – try to, doesn’t work)
-rw-r–r– 1 web-user web-user 4182 Nov 21 12:07 known_hosts
Owner is okeey everywhere.
Key was added to deployment-keys
Post hook was added with right url-path
($payload = json_decode(stripslashes($_POST[‘payload’])); – i tried to
I still have problems with payload, what is reason?
line 13: $payload = json_decode($_POST[‘payload’]);
line 36:
file_put_contents(‘deploy.log’, serialize($_POST[‘payload’]), date(‘m/d/Y h:i:s a’) . ” Deployed branch: ” . $branch . ” Commit: ” . $commit_hash . “\n”, FILE_APPEND);
My log:
N;12/01/2014 02:12:15 pm Deployed branch: Commit: c7e81bc
N;12/01/2014 02:16:50 pm Deployed branch: Commit: c7e81bc
N;12/01/2014 02:16:50 pm Deployed branch: Commit: c7e81bc
N;12/01/2014 02:16:51 pm Deployed branch: Commit: c7e81bc
Php warnings:
PHP Notice: Undefined index: payload in /var/www/web-user/data/www/dentist.web-user.com/deploy/bitbucket-hook.php on line 13
PHP Notice: Undefined index: payload in /var/www/web-user/data/www/dentist.web-user.com/deploy/bitbucket-hook.php on line 36
PHP Notice: Undefined variable: branch in /var/www/web-user/data/www/dentist.web-user.com/deploy/bitbucket-hook.php on line 36
PHP Notice: A non well formed numeric value encountered in /var/www/web-user/data/www/dentist.web-user.com/deploy/bitbucket-hook.php on line 36
PHP Warning: file_put_contents() expects parameter 4 to be resource, integer given in /var/www/web-user/data/www/dentist.web-user.com/deploy/bitbucket-hook.php on line 36
How to push to Bitbucket? Inside .git or outside?
@Toan You do any development on your local machine and push changes to your Bitbucket repository using `git push`. If you’re not sure how to do this you should consult the Bitbucket documentation.
Thanks for this great guide as I’m new to Bitbucket/Git. I am stuck at the GIT_WORK_TREE step. When I run that command, it says ‘did not match any files(s) known to git’. When I check the branches folder inside my .git repo, it is empty. Why? I had created a production branch from my master branch before starting all this. I didn’t make or commit any changes to the production branch, as I would like to work in master and then merge changes with production. If I understood correctly, this is what you are doing as well.
Hey Great tutorial and got up to speed and running fairly easily. One little snag though. It seems like the –mirror command didnt work for me. The .git on my server is not updating with the most recent one. Did i miss something on this?
I noticed that when I visit my php file in the browser i get an internal server error i am guessing this might be the issue. It has the following error:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@jacobroman.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Any suggestions???
Hi, I am quite a newbie. Your blog is helpful.
But the second line gives me an error.
cd ~/.git
GIT_WORK_TREE=/home//www git checkout -f production
The error says, fatal: This operation must be run in a work tree
Hi,
Thanks for this excellent post. Worked very well for me!
Rahul
Works like a charm! (just had to switch away from my master branch and start checking out other branches)
Nice post! Thanks for taking the time to write such a clearly explained method to achieving this – just what I was looking for.
Happily deploying test code on a server on every push now. Result! :-)
Good job Jonathan. Thanks for this.
Worked, but update my depoy.log file only
I am a newbie in GIT,
I have made both of repo & www folder to be owned by www-data user
So how to know where is my problem exactly?
Thank you, Jonathan!
@Ardi – I had a similar problem. Check if you really created ssh files for the www-data user. What’s crucial is to run at least first pull as a www-data user on the cli. Why? Because you have to add bitbucket’s RSA to the list of known hosts.
So run “sudo -u www git pull” and you should get something like this http://d.pr/i/1e8SR
@Lucien144 It works!!
So, finally I just setup SSH keys in /var/www/.ssh for www-data user, since there is no /home/www-data :D
Thanks very much
Hi.
Great post – thanks! I’ve set it up on a couple of servers with no issues at all.
In regards to “security through obscurity”, you’re probably better off obscuring the deployment folder name rather than the deployment filename.
So instead of creating a folder “deploy” create the folder “a13jsur5kcidwe89z” and your deployment hook url becomes http://<domain>/a13jsur5kcidwe89z/bitbucket-deploy.php
As long as you put the right folder structure into BitBucket as the posthook then there are no other changes.
But you’ll be less likely for somebody to trollscan for /deploy/deploy.log
Cheers
Jules
Hi Jonathan, great article.
Thanks for taking the time to write this.
However,I’m facing the same issue as described by BK: encountering an
“error: pathspec ‘….’ did not match any file(s) known to git.”
when doing the
“GIT_WORK_TREE= git checkout -f ”
I found some suggestions this might might be fixed by doing a git fetch first, but this does not resolve the issue for me.
Any ideas on how to solve this?
One more thing which helped me out and might be added to the article was Mick O’Hea ‘s suggestion:
—
If you’re creating the SSH config file because it doesn’t already exist, you need to change the permissions on it with a “chmod 600 config
In my case, this was needed at the top of my script :
exec(‘eval $(ssh-agent -s)’);
exec(‘eval “ssh-agent ssh-add id_rsa”‘);
And this at the end :
exec(‘ssh-agent -k’);
Just a word of advice to any that may be attempting to implement this method, if you suspect that your exec()’s are failing and are having trouble diagnosing your setup, try executing the commands as the user you have configured from the command prompt.
In my case, I configured www-data user to do the git fetch, so use “sudo -u www-data git fetch” then you’ll get feedback as to why your fetch is failing. Once I tried that, a quick adding of known_hosts, and ensuring to do a chmod 600 on the rsa and config files solved my problem.
Thanks again for the brilliant tutorial!
Thanks for the post, Jonathan. I have a question: using the steps above, would I be able to pull changes that were made to the live site from the server into my Bitbucket repo?
webhook is a lightweight configurable tool written in Go
– https://github.com/adnanh/webhook
Hi Jonathan,
thanks bigtime for this nice tutorial. It s exactly what i was looking for.
However i cannot get it to work with the webhooks from bitbucket. If i push changes from my local working copy, the remote repo gets updated but the webhook gives me the 404 error. So communication with my server was established but the script was not found.
This is the URL I tried: `http://ip.ip.ip.ip/home//app/deploy/bitbucket-hook.php`
which matches the path to the script.
I updated the script variables matching to my needs and the ssh keypair (without password) is also working because i can clone the repo manually. I think i am good there. Does someone has some suggestions on this ?
Cheers
Tim
It would be great if you give us a hint what automatic deployment you switched to!
@Florian I have a lot of clients on shared hosting accounts, and for them I use git-ftp.
Just a heads up:
retrieving the payload as described in the php script (bitbucket-deploy.php) does not work anymore for Atlassian bitbucket (at least not for me).
To get the payload use something like:
$request_body = file_get_contents(‘php://input’);
$payload = json_decode($request_body);
The contents of the payloads are described here:
https://confluence.atlassian.com/display/BITBUCKET/Event+Payloads
@Pascal: You are right. Atlassian recently changed the way how webhooks work.
I followed the whole procedure religiously. In the End , The web hook made doesn’t works , and shows a 502 error. I dont know but may be Atlassian made some changes , which doesnt allows you to hook in normal way . The deploy.log is still empty due to that error.
@Pascal: Thanks a lot for the update! Using $_POST is outdated.
Hi, thank you for the post, can I use ruby-git on my server? I tried to find git on my server, could not find it. It has capistrano though as a ruby gem; can I use capistrano or ruby-git for automating deployments?
Thanks.
Plus one more thing, how can I install git if ruby-git can’t help.
Thanks.
Can someone please explain how the webhook is setup now that POST does not work. In BitBuckets Settings>webhooks what is the trigger I should use? I presume repo push?
Used Jonathan’s idea and code for creating more advanced version. Applied new bitbucket’s webhook interface. Added ability to deploy multiply projects via one script.
Code located on my bitbucket:
https://bitbucket.org/lilliputten/automatic-bitbucket-deploy/
Many thanks, I used your post and it works !
Great article, thank you! Auto-deployment is a very satisfying workflow. If anyone is worried about security on their webhook, Bitbucket offers a list of IP addresses that official requests will come from. A simple check at the top of the PHP script could accept Bitbucket requests and drop nefarious ones.
any reason why when I commit a change it doesn’t overwrite any of the files that were there previously? If the project directory is empty, it works fine, but not if the files were already there and I’m trying to update those files. I’m at a loss as I see using ‘checkout -f’ should overwrite them and it is being used in Igor Lilliputten’s bitbucket.php file but it obviously isn’t overwriting. Thanks!
This means i have to make :
cd ~/.git
GIT_WORK_TREE=/home//www git checkout -f production
each time i want to update my site ?
Hi,
If some body need to know ;)
In ubuntu the home folder for user www-data is /var/www so the id_rsa must be at /var/www/.ssh.
You can create with
sudo -Hu www-data ssh-keygen -t rsa
And after you copy the key to bitbucket or github you muts to do a first manual connecto to the server.
sudo -Hu www-data git clone git@bitbucket.org:username/repo.git
This is because a know_hosts is needit
The authenticity of host ‘bitbucket.org (104.192.143.3)’ can’t be established.
RSA key fingerprint is XX:XX:XX:XX:XX:14:xx:5c:3b:ec:aa:46:46:74:7c:90.
Are you sure you want to continue connecting (yes/no)? yes
Max ,
When try to create the key, I get something like:
” open /var/www/.ssh/id_rsa failed: Permission denied.”
Why is this?
Hello
I am not sure, but I think the user www-data it is not really the owner of /var/www/.
You can change that with
sudo chown www-data:www-data /var/www/
and try again.
Do you have privilege of sudo in your account?
It is a ubuntu or debian OS?
look this http://ubuntuforums.org/showthread.php?t=1615133
This post really helped me out, thanks for that!
I thought letting PHP/Apache to run git (therefore giving it access to my ssh keys) was somewhat insecure, so I decided to try something else.
Instead of using a webhook I changed that php script to a shell script that runs every minute to check for updates. It’s been running for months and is working like a charm. :)
If anyone wants the file (or if you want to add it to your post, Jonathan), just drop me an email.
On my hosting disable all “exec” function of php.
I use this dirty solution for autodeploy:
replace in php script last if
if ($update) {….}
on this
if ($update) {
// Do a git checkout to the web root
file_put_contents(‘./go_go’,’GO!’);
file_put_contents(‘deploy.log’, date(‘m/d/Y h:i:s a’) . ” Deployed branch: ” . $branch .”\n”, FILE_APPEND);
}
added next bash script to crontab – execute every minute
(path in variables need change)
#!/bin/bash
secs=58 # Set interval (duration) in seconds.
FLAG_FILE=”/home/werty/public_html/_deploy_3434d5345/go_go”
SECONDS=0 # Reset $SECONDS; counting of seconds will (re)start from 0(-ish).
while (( SECONDS < secs )); do # Loop until interval has elapsed.
sleep 1
if [ -f "$FLAG_FILE" ]; then
rm -f $FLAG_FILE
cd /home/werty/dservice.git
git fetch
GIT_WORK_TREE=/home/werty/public_html git checkout -f master
git rev-parse –short HEAD
fi
done