This post is primarily for my future self to document how to deal with the problems I had installing Octopus Deploy using a domain account as the service account.
I am currently working with a team that has old versions of TeamCity and Octopus Deploy and want to move to the latest versions. The upgrade path from these (very old) versions to the latest versions is complicated and therefore we have set up a new server to host the latest versions.
The (virtual) server we will be installing on is hosted in a managed environment (with no self-service facilities). This makes taking snapshots/checkpoints to rollback to if things go wrong during installation a bit more complicated. It also means that we effectively only get one shot at getting things right.
For these reasons, I created a small test lab at home to practice all the steps required so that I could create an installation step-by-step 'run-book' to do the real installation with as few hiccups as possible.
In my test lab I have the following set up as Hyper-V virtual machines:
- A Windows 2016 server that is the Active Directory Domain Controller and hosts a SQL Server 2017 database server
- A Windows 2016 server that doubles up as the TeamCity and Octopus Deploy server
- A Windows 2012 server that will be a target server for deploying web sites to via an Octopus Deploy Tentacle.
This three server set-up does not truly represent a real-world environment (for example, the SQL Server would not normally reside on the domain controller), but is enough to emulate the core elements of the environment that I will be using in the real world.
The previous installation of Octopus Deploy used a local Windows system account, but for the new installation, we want to use a domain account
- to connect to SQL Server using Windows Authentication and
- to have tight control over what the service can do vs the local system account which is tantamount to having full local administrator rights.
At time of writing, the details are a bit vague in places and require further information.
Based on this, I went through the following actions.
- Create the Windows domain account with password locked down so it can be used as a service account
Then on the server that Octopus Deploy will be installed on:
- Create a folder called 'Octopus' on the D: drive for the Octopus Deploy application to reside in and grant Full Control to the Octopus service account created in AD
- Create a folder called 'OctopusData' on the E: drive that will be used as the Octopus 'home' location and grant Full Control to the Octopus service account created in AD
- Grant the Octopus service account created in AD the Log on as a Service permission.
Details of the requirements for the SQL Server version and creating a database can be found on the Octopus Deploy website but in summary, if the SQL Server is not on the same server as Octopus, the following needs to be done before installation starts.
- Ensure network connectively to the server over TCP/IP
- Ensure no firewall rules preventing access over TCP/IP on appropriate ports between the server hosting SQL Server and the server hosting Octopus Deploy
- Create an empty database in SQL Server 2017 called Octopus
- Set the collation of the database to case insensitive using
ALTER DATABASE [Octopus] COLLATE Latin1_General_CI_AS
- Create a SQL Server login for the Octopus service account created in AD
- Create a user in the Octopus database for the Octopus service account login
- Set the default schema for the user to [dbo] so the account has full control over the database.
At this point there are some other things that I should have done, but did not find out until I had problems further down the line.
To view these next steps, scroll down to here.
Starting the installation is simple enough, with just one change from the default option, and that is to change the installation folder to D:\Octopus (where permissions have already been granted).
On clicking 'Finish' there is a brief pause while PowerShell is checked and then the 'Getting Started' Wizard is opened.
After entering the licence key on the first page, the next step is to the change the 'Home' directory to the E: drive where there is plenty of disk space.
Clicking next to move to the next page, it is time to select the service account.
By default, this is set to the Local System account, but as described above, we want to use a custom domain account, so we change the radio button option and enter the domain credentials (using the Select User to find the service account created in Active Directory).
The next step is to point to the Octopus database that has been created in SQL Server 2017 and for which the service account has been granted DBO rights so that the database tables and other artefacts can be created.
The next screen prompts for a port for the web portal. As the server that I am installing on has IIS, TeamCity and a couple of other self hosted web applications, it is not appropriate to use the default port 80. In the example below I am using port 93.
This is one of the areas where things started to go wrong later down the track!
Moving on to the next page, this is where the administrator for Octopus Deploy is selected.
This is the next place that there are a few gotchas!
So the screen looks innocent enough. When you first arrive on the screen , it defaults the Authentication Mode drop down to using usernames and passwords in Octopus. However, to make life simpler, we want to use Active Directory. When this is selected, the username will default to Administrator.
DO NOT JUST ACCEPT THIS! This will be the local Administrator account for the machine you are installing on.
Instead, use the Select User link to bring up the standard Windows dialog for selecting a domain user.
As I was installing in a test lab, I was a bit lazy and just selected the Domain Administrator account. DO NOT DO THIS EITHER!
The problem that you will find (after installation is complete), is that, by default, the built in Domain Administrator account does not have a User Principal Name (UPN) and without a UPN, Octopus cannot find the account in Active Directory.
When this happens, you get a "No fallback name was provided" error when trying to login.
At that point, you are pretty much locked out of Octopus and either have to add a UPN to the Domain Administrator account or re-install Octopus from scratch to use another user as the Octopus administrator (though you may be able to change the authentication provider through the command line (I haven't tried this! See https://octopus.com/docs/octopus-rest-api/octopus.server.exe-command-line/configure. )
As this can cause a lot of head-scratching, it would be nice if the installer could do the checks to verify whether the account selected will work at login stage before proceeding to the next stage.
Instead, depending on your team setup, you should select your own domain account from the dialog, or (if you have the right AD permissions) create a dedicated account for this. This second option adds complexity as you will need to log in to Windows using that account to make full use of Active Directory integration, or (if using Chrome for example), enter the credentials in using Forms Authentication or via a browser dialog box.
To get around all this I changed the account to my domain account with a view of adding other administrators later.
Another feature that would be nice here is to be able to make an AD group the administrators rather than one specific user as it is all too easy to forget how this has been set up when that person leaves and their AD account is deactivated.
Clicking Next takes you to the final page to click Install and get ready to get up and running with Octopus Deploy.
Except ... things did not go according to plan!
Looking at the both the Windows Event Log and the Octopus (text) logs , I found a couple of occurrences of this
Microsoft.AspNetCore.Server.HttpSys.HttpSysException (5): Access is denied.
In short, whilst I had prepared the basics of granting file system permissions to the service account, I had failed to grant permissions to access the HTTP traffic over the ports that Octopus needs to monitor.
I had made the mistake of assuming that the installer would take care of this, not having really understood the prerequisite notes on the web site.
This is where I found help on the Octopus website at https://help.octopus.com/t/octopus-server-not-starting-httpsysexception-from-microsoft-aspnetcore-server-httpsys-urlgroup-registerprefix/25025/3.
In that post, it explains that the NETSH command is required to provide permissions to HTTP access on certain ports.
NETSH http add urlacl url=https://+:port/ user=domain\username listen=yes
An explanation of the syntax can be found at https://docs.microsoft.com/en-us/windows-server/networking/technologies/netsh/netsh-http#add-urlacl, however that is still a bit dry and didn't really explain what the problem was.
A bit of searching lead me to this Stack Exchange answer https://superuser.com/questions/1272374/in-what-scenarios-will-i-use-netsh-http-add-urlacl that made things a bit clearer and in turn pointed to this Microsoft explanation https://docs.microsoft.com/en-us/windows/win32/http/namespace-reservations-registrations-and-routing?redirectedfrom=MSDN.
So, here's the fix that is required to address the Access Denied error.
In short, you have to grant the permissions to the service account to listen for HTTP requests on certain ports.
To do this, open a Command Prompt on the Windows Server you will be installing / have just installed on using Run As Administrator and enter the following command for each of the ports required, substituting port_number with the actual port number required
netsh http add urlacl url=http://+:port_number/ user=domain\username listen=yes
In my case, as I am using port 93 instead of 80
netsh http add urlacl url=http://+:93/ user=MyDomain\Octopus listen=yes
netsh http add urlacl url=http://+:10943/ user=MyDomain\Octopus listen=yes
The entry for port 10943 is so that Octopus tentacles can poll the server. See https://octopus.com/network-topologies for details.
This blog is the result of an afternoon of re-applying Hyper-V snapshots and installing Octopus (it took seven attempts in all) until all the quirks I describe above had been ironed out.
I must admit, I am surprised that given that the installer knows the ports and the service account, that it can't do this work for you. Same goes for checking the administrator account meets requirements such as having a UPN.
Perhaps it may be a UAC elevated rights problem as it seems to be a common thing to have to do (try searching for "NETSH http add urlacl" and you will find results for various software packages that have this manual step documented), though this is the first time I have had to address this problem manually.
Don't get me wrong - I love Octopus Deploy, but as this was my first time installing it from scratch, having to go through these hoops to get the latest version installed and running as I need it did test my patience.