Continuous integration and deployment automation

A casual discussion to start us off. ;) I would be interested to know everyone here does CI and deployment automation with Sitecore - particularly if you have run large projects with multiple CD and CM environments. We don't have much written down on the subject and I would love to write a bit of a guide. So:

  • What does that process look like?
  • What tools do you use (TDS, powershell, TeamCity)?
  • Are you on Azure / AWS or on-prem?
  • How often do you deploy?
  • Can you outline the general steps?
  • Are there any big 'gotchas' that we should be aware of?

43 Replies

  • We're using a very simple process, only to dev and test so far. I don't know if we'll ever get to the point where we'll deploy directly to prod, here it's more a political decision than a technical one.
    We basically are using TFS and MSBuild along with TDS to package things up correctly, then things just go to a mapped drive to the various environments we have.
    All equipment is internalish....... we use Dell hosted servers, but have our own admins for them.
    We're supposed to be moving to using Bamboo as our build server, no idea what their timeline is.
    Anyone have any experience with that?
    Hi everybody!
    Mike
  • In reply to Mike Mormando:

    Hi! :) Thanks for sharing.

    • When you deploy to test/dev, do you tear down everything and deploy all items again (templates/layouts/config as well as content) or merge in changes?
    • When you upgrade, do you maintain the content on dev/test or start again?
    • How quickly can you see your changes on test?
  • In reply to Martina Welander:

    We've got a pretty large content team that works strictly in production at this point. Periodically we move that content down to the test server so we get a more realistic view of what we're seeing when the QA team hits test. What gets deployed it basically determined by version control, Stash/Git in our case. When we check things in, the joint dev environment is deployed to automatically. After we have had a chance to look at that, we can then manually kick off a test TFS build that will rebuild with any set changes needed for test, then deploys that, so about 10 minutes of so, determined by how fast the project builds.
    ;)
  • Our Process at Branded3 currently works like this....

    we have 4 environments; continuous delivery, internal UAT, external UAT and production.

    code is checked into github which includes the Sitecore root and our code (I know this is against best practice but it works well for software assurance, maybe a nuget feed would be good but I havent looked into it yet)

    we then use team city to build the solution and run any gulp tasks for SAAS/js minification and finally run octopack to package up all the files into what is basically a nuget package.

    we then use octopus deploy to automatically push the code out to continuous delivery. When we get to a milestone build, where we are considering pushing out to production we will use octopus deploy to push the package to internal UAT for in house testing, then to external UAT for client testing. if all is good then it goes to production.

    we are using unicorn for item serialisation and these files get published into the package, when we deploy we manually run the sync command (this will soon be automated)

    when the build gets pushed to each environment we run a post deploy powershell script which copies over any environment specific configs from a folder included in the package and repoints the IIS directory to the new location

    we also have one project which uses the Sitecore Azure module and for that we use octopus deploy the same way but the production environment pushes only to the admin server then we log in once the release is complete and use the controls in Sitecore to deploy to the CD environment although soon i will be replacing this with our own powershell deployment so that it can all be done in one step by octopus deploy

    The biggest gotcha I have found is with the way Sitecore does upgrades, it is totally at odds with the modern deployment mechanism so if you want to upgrade to a new version of sitecore you have to run the upgrade process on each environment individually and then push the updated build, otherwise you don't get item changes. The only other solution is to serialize the entire core and master database but that isn't a good idea. 

  • In short:

    * TDS for local deployments
    * TDS for serialization and packaging
    * Sitecore.Ship for delivery to remote servers
    * Elastic Bamboo as a CI engine
    * Sitecore.FakeDb as a primary unit tests isolation technique
    * Powershell for everything that is not MSBuild + TDS (personally like writing PS scripts better than building custom MSBuild tasks)
    * Curl for HTTP orchestration (personally like it better than doing the same in PS)
    * NuGet for distribution (with custom init.ps1 to automate deployment to local upon package install)

    A little more visual in our How SCORE Is Made video: www.youtube.com/watch

    Also ask (deleted user). He recently had a chance to see it all up close.
  • At we have following:

    • Git as a source control (TFS for some client - and it really do note recommend it.)
      • Branch per environment (CI, QA, UAT, PROD)
      • feature branches
    • TDS for serialization, packaging and local deployment
    • TeamCity / TFS (may vary per client)
    • Deployment to CI on change, QA every night, UAT & PROD per request
    • Steps different for each environment
      • e.g. DEV
        • Clean web root + db
        • restore Nuget
        • deploy filed
        • deploy items
        • smart publish
        • run tests in web db container
        • republish
        • reindex
        • merge to QA on success
    • Stages may vary depending on amount of environments (e.g. working with swapping cores in SOLR, taking off/in load-balancer and so on.)
    • LESS scripts complication via bundles
  • Hi,

    We have typical 4 environments: local development machine, DEV, ACC and production. Source control includes our code and the most important sitecore items. On each push, the build server pulls the latest, compiles and validates. We have nightly deploys to the DEV environment. Deployment to ACC is manually triggered. No automated deploy to production environment yet.

    The automated deploy process looks as follows:

    • Pull latest code and compile
    • Deploy with msdeploy
    • Create diff package of sitecore items against previous deploy (custom tool). The package is installed with Sitecore Ship.
    • Tag code

    Tools used:

    • Slowcheetah (and publish profiles) to prepare the config files for the various environments.
    • Jenkins (moving to bamboo shortly)
    • Gitlab (moving to stash shortly)
    • Unicorn to get items serialized
    • Custom tool to generate diff package of serialized items
    • Sitecore Ship to deploy diff package

    Kind regards,

    Thomas

  • After a few attempts, I like the following:

    - TDS & Glass – Auto generation models
    - TDS – serializing items for source control
    - Slow Cheetah – config transforms
    - Team City – Build server
    - Sitecore Powershell – Install Sitecore modules
    - NuGet – deployment packages to AWS / Azure
    - Octopus Deploy – Deploy Packages to servers

    For local developer machines SIM (Sitecore Instance Manager) is great.
    Octopus deploy latest release has “deploy to Azure PaaS”, which is great too. And worth considering.
  • A lot of my ideal deployment process stems from the solution structure which I describe at a high level here: cardinalcore.co.uk/.../sitecore-solution-structure

    I have worked with a lot of different set ups, so will just describe how I *choose* to do it :D

    What does that process look like?

    What tools do you use (TDS, powershell, TeamCity)? TDS, TeamCity, Octopus, Sonar, Myget / Proget

    Are you on Azure / AWS or on-prem? On premises generally

    How often do you deploy? I would choose to deploy to dev on every commit, QA / Production are governed by applying tags in Git

    Can you outline the general steps? Build > Run Tests > Create Nuget Package > Pass to Octopus for Deployment > TDS / Unicorn > Publish

    Are there any big 'gotchas' that we should be aware of? TDS / Code Deploy can sometimes be the wrong way round to guarantee deployment working. TDS must be the same version on build server and production. Git CRLF handling can cause issues with serialized content
  • In short:

    What does that process look like?

    1. Write an amazing code :)
    2. Check it in to TFS (after code analysis)
    3. Run the automated build (which will execute the unit tests)
    4. Run some kind of http crawler to check if everything is updated correctly.

    What tools do you use (TDS, powershell, TeamCity)?

    1. TDS
    2. Sitecore SHIP
    3. TFBuild
    4.  Powershell if necessary
    5. Sometimes I use RAZL scripts to auto-sync content

    Are you on Azure / AWS or on-prem?

    On premise - most of the time.

    How often do you deploy?

    On Dev - whenever you feel it. QA - twice a week. Prod - once a week.

    Can you outline the general steps?

    - TDS Package Serialization

    - Modified SHIP for package installation.

    - TFBuild (Don`t hate !) because we are still using TFS :) + MSBuild tasks to post the packages to SHIP

    - Slow Cheetah - for config transforms

    - NuGet for package distribution

    Are there any big 'gotchas' that we should be aware of?

    I am still old fashioned and I prefer to do manual deployments of TDS Packages on Production :)

  • This is brilliant. Thank you everyone that has contributed so far. Does anyone feel like doing a little video of any part of their process..? :) Maybe your TeamCity configuration? In fact - if anyone has the time or inclination to talk me through a full deployment (on video, for general distribution) I think that would be incredibly valuable.
  • In reply to Martina Welander:

    i was thinking about doing a series of blogs on this. would that do? I could maybe guest post on the sitecore blog
  • We use JIRA+BitBucket+TeamCity+FTP.

    • JIRA for change management (issue tracking)
    • BitBucket for source control (tied to our JIRA so issues are tied to commits)
    • TeamCity to pickup specific branches
    • FTP plugin to upload serialization files (stored in source)
    • Azure for development environment

    Our TeamCity configuration is pretty straight forward.

    1. We listen for changes on any build-* and dev-* branches.
    2. Copy config for the specific deployment (<origin>/src/config/dev) into the source director (<origin>/src/<Sitecore Project>)
    3. Build the solution
    4. Build and WebDeploy the specific <Sitecore Project>
    5. FTP up the serialization files (<origin>/serialization)
  • Hi, Teamcity+Git, here are steps:
    Step 1: Insert app_offline
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: same as checkout directory
    Custom script: view script content
    Step 2: Restore DB
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: same as checkout directory
    Custom script: view script content
    Step 3: Clean webroot
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: same as checkout directory
    Custom script: view script content
    Step 4:
    Runner type: NuGet Installer (Installs and updates missing NuGet packages)
    Execute: If all previous steps finished successfully
    Path to NuGet.exe: NuGet (version default (2.8.2))
    Package Sources: Use nuget default package source
    Path to .sln: source/some.sln
    Exclude Version: OFF
    Restore Mode: Restore packages (requires NuGet 2.7+)
    Use local machine packages cache: OFF
    Update packages: OFF
    Update mode: Update via solution file
    Use safe packages update: OFF
    Include PreRelease packages: OFF
    Step 5: Deploy: Website
    Runner type: Visual Studio (sln) (Microsoft Visual Studio solution (.sln) runner)
    Execute: Only if build status is successful
    Solution file path: source/some.sln
    Working directory: same as checkout directory
    Visual Studio: Microsoft Visual Studio 2013
    Targets: Build
    Configuration: %ci.configuration%-Website
    Platform: default
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: same as checkout directory
    Custom script: view script content
    Step 7: Publish: TDS
    Runner type: Visual Studio (sln) (Microsoft Visual Studio solution (.sln) runner)
    Execute: Only if build status is successful
    Solution file path: source/some.sln
    Working directory: same as checkout directory
    Visual Studio: Microsoft Visual Studio 2013
    Targets: Publish
    Configuration: %ci.configuration%-TDS
    Platform: default
    Command line parameters: none specified
    Step 8: Smart publish
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: same as checkout directory
    Custom script: view script content
    Step 9: Test Runner
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: same as checkout directory
    Custom script: view script content
    Step 10: Full Republish
    Runner type: Command Line (Simple command execution)
    Execute: If all previous steps finished successfully
    Working directory: Full Republish
    Custom script: view script content
    Step 11: GIT: Automatic PUSH to DEMO branch
    Runner type: Command Line (Simple command execution)
    Execute: Only if build status is successful
    Working directory: same as checkout directory
    Custom script: view script content

  • In reply to Konstantin Cherkasov:

    I've used TFS and Cruise Control for the CI, MSBuild for the build server, TDS for content serialization and deployments, PowerShell for post-deployment steps, then custom service calls for package deployments and house keeping like publishing, reindexing...etc. About to set another CI solution with Jenkins.