Azure role recycle – What the hell

It was a long night – one lasting 27 hours, actually. It was the moments before deploy. Your team is counting on you. Your customer is counting on you. You are counting on you. You hit deploy. You wait. You pray. You’ve cut some corners to make the deadline, but you hope it won’t show. And then the unthinkable happens. Azure role starts to recycle. Now what!?

I tried everything I’ve known up until that point. I’ve ran the solution locally in an emulator, and it worked. As I found out later, that had a logical explanation as well. I went through failed IIS request logs, I went through Event Logs. Mind you, this was a challenge because our IT department has, for some reason, blocked outgoing ports as well :-). But that’s a different story.

The only symptom we could see was “Role is restarting…” in the Azure Management web site. When IIS logs were empty that’s when I started to suspect something else was going on. After a lot of searching, I ran into this link:

Following the instructions on the site and links on it, I RDPed to the VM in Azure, and ran the following in a PowerShell window:

From there on, the rudimentary looking, but very useful Azure Tools was downloaded and ran. I then downloaded X64 Debuggers, then on the Utils tab, I use the Attach debugger…

“This is arguably one of the most useful utilities in AzureTools. This will let you attach a debugger to a process which fails and exits immediately on startup.  The most common scenario is a role which is recycling and you want to attach WinDBG to the WaIISHost/WaWorkerHost process, but it crashes too quickly for you to manually attach the debugger.  In Azure the normal trick of setting the Image File Execution Options debugger registry key doesn’t work and this Attach Debugger utility is the only consistent way I have found to attach a debugger.  To use, first download a debugger (ie. double-click the X64 Debuggers And Tools-x64_en-us tool from the Tools tab), enter the process name you are interested in, click Attach Debugger, and wait for the Azure guest agent to automatically start that process again.”

If you have never used WinDbg, make sure to read an article or two before doing so. This is a good start. I was really lucky as the process actually threw an exception that was not being bubbled up to the Azure host – so I didn’t see any information about it.

This is what I got in the WinDbg window:

From here on out, things were pretty clear! We weren’t using a RoleEntryPoint and the exception was happening very very very early on (think init inside Global.asax). The underlying issue with us was that we had an invalid binding redirect in the web.config. It was invalid because we are using NANT to build the web.config file (long story there…), and someone didn’t update the new versions after updating the NuGets. It worked locally because the old DLL was still somewhere in cache.

I haven’t been able to reproduce the issue properly just yet, but I will try again soon. When I succeed, I will update this post. Until then, I hope the links I found can help someone struggling with the Role Recycle problem.

Anže Vodovnik is a proud father, enthusiastic guitarist and passionate software developer. He enjoys presenting at conferences sharing his experience of over 15 years of creating software. He was briefly a Microsoft MVP for Azure before forfeiting the title when he joined Microsoft UK, where he’s now working hand-in-hand with customers to help them develop and use solutions based on the Microsoft Azure platform.

View Comments