Octo.exe – exporting/importing project pitfalls

Pitfall_Coverart Over a year ago, I mentioned that the Octo.exe supports the ability to export a project from one server to another server. This feature last year needed a lot of work and was a bit difficult to understand, but at the time I didn’t have a pressing need for it. After working with it extensively over the past couple of weeks, I’m confident in saying the Octo.exe can definitely export/import properly, but there are some caveats and pitfalls you need to be aware of. Once you realize this potential headaches, you’ll know what it takes to move your project from one server to another.
Currently our Octopus Deploy server is overloaded with projects and users. Many months ago, we made the decision to break up the Octopus Server into smaller more manageable projects across Octopus servers so the export/import feature within the Octo.exe became a very big help in accomplishing that goal. While we’re still in the process of moving projects, I learned quite a bit about the export/import functionality and why it’s somewhat tricky to work with.
First, when trying to move a project to another server you have to realize there are number of dependencies that exist within a given project and it’s variables. These dependencies include:

Project Group
Environments
Machines
NuGet Feeds
Variable Sets
Step Templates* – FYI, the Octopus Documentation doesn’t mention this (check the prerequisites section) – any step templates you’ve used from the library needs to be imported before hand

These need to be present on the server (with the same names) where the project is being imported to. Why? Well, all of these objects are intricately linked together. If one of these pieces where missing, how is the new Octopus Server supposed to handle the missing piece? Once you acknowledge this, the process becomes easier to understand.
The basic workflow for exporting and importing a project is simple:

~~Log onto the Octopus Server where the project currently resides~~
From the command line, run the octo.exe export command and generate the exported json file.
~~Log onto the Octopus Server where the project is moving to and copy the exported json file~~
From the command line on the new server, run the octo.exe import command using the exported json file as the input
Magic happens and everything looks like it did before (hooray!)

Edit: You don’t have to log into each server – you can do everything from your workstation as long as you can access both servers via the network. Depending on your network configuration, this may vary for you but just know that it’s not required to actually run the octo.exe in the octopus server itself. It can all be done remotely.
After doing this process several times, I ran into a few snags along the way. I was able to overcome all the obstacles, but had I not thought about the dependencies before hand, it would have been a much rougher experience. Fortunately for us, we have an automated provisioning system that builds our project groups, environments and NuGet feeds. Scratch three off the list. Our teams don’t use variable sets so that leaves two issues to worry about: Machines and Step Templates.
But wait, what if you’re importing a project that’s being moved to a different project group? Well, that doesn’t work unfortunately. Luckily, you can manually edit the exported json file! Within the exported json file, search for the string: “ProjectGroup”. Here’s an example:

"ProjectGroup": {
    "Id": "ProjectGroups-234",
    "Name": "MyOldProjectGroupName"
},

Simply edit the “Name” within the Project Group data. Don’t worry about the Id field as the importing Octopus Server will correctly match the valid Id as long as the New Project Group name exists on the Octopus Server.

"ProjectGroup": {
    "Id": "ProjectGroups-234",
    "Name": "MyNewProjectGroupName"
},

So even if your project that’s being imported is moving to a different project group, you can do it with a little editing of your exported json file.
Okay, now were down to the final two dependencies: Machines and Step Templates. Step templates are straightforward – as I mentioned earlier, whatever step templates you used on the prior Octopus Server have to be installed on the Octopus Server you’re importing into. That leaves us with Machines. Oh… machines. If you’re using machines in your variable scopes, prepare for some pain.
Now one scenario we have yet to test is moving tentacles to the new Octopus Server before importing the project. Most teams are unwilling to do something like that which is seemingly odd. However, given that some of our projects have 10, 20, even 30 or more tentacles, I can understand the reluctance to do such a move. So up to this point, I’ve been migrating projects first and then we move over the tentacles. This can be problematic, especially for projects that use Machines in their scopes for variables. And when you think about it, no wonder the Octopus Server has to reject the import – there’s no way for the Octopus Server to know what machine you’re referring to.
So if you’re looking to migrate your project before moving over any tentacles, you have to edit your exported json file. There’s no way around this.
Within your json file, you can search for Machines directly:

{
   "Id": "1c9c86ef-dcdd-5bce-1024-ebc4e71ec3c0",
   "Name": "MySuperCoolVariable",
   "Value": "I am so l33t",
   "Scope": {
   "Environment": [
      "Environments-102"
   ],
   "Machine": [
      "machines-1024"
   ],
   "Role": [
      "WebServer"
   ]
   },
   "IsSensitive": false,
   "IsEditable": true,
   "Prompt": null
 },

Search for “Machine”: [ should take you to each instance of a Machine being used in the scope for any variable. Simply hack out the Machine sections for all of your variable scopes. It’s a terrible hack, but it works and we can move over teams after importing the project. Note: I should be using PowerShell and regular expressions to do this. While I’m proficient in PowerShell, I can barely manage my way through regular expressions. If I ever have time, I’ll come up with a PowerShell script (using regular expressions) to strip out these machines within a json export file.
In reality, the machine scope for variables is really overkill and you should avoid if at all possible. I can imagine in rare cases where it’s necessary, but as I’ve mentioned in my Variables Deep Dive, being too explicit with your variable scope will forever force your hand to always be explicit – even if it’s not necessary. I’d argue for most of the teams I’ve imported, 95% of all the uses of machines in their variable scope were completely unnecessary. So here’s the resulting json file segment:

{
   "Id": "1c9c86ef-dcdd-5bce-1024-ebc4e71ec3c0",
   "Name": "MySuperCoolVariable",
   "Value": "I am so l33t",
   "Scope": {
   "Environment": [
      "Environments-102"
   ],
   "Role": [
      "WebServer"
   ]
   },
   "IsSensitive": false,
   "IsEditable": true,
   "Prompt": null
 },

Simply removing the Machine from the variable scope will allow you to import your project without having the servers moved over to the new Octopus Server. For most of our teams, its fine because they never really needed the machine scope anyway, so in a small sense we’re refactoring their variable scope and migrating them at the same time. Well, that’s what I’d like to think anyway. I hope you found this Octopus Tip useful! I’ve got more deep dives and other articles coming soon!

Categories:Octopus DeployTip