All posts by wichita.sao

My frustrations with the Distributed Cache

Archive, SharePoint

The distributed cache was a great idea introduced with SharePoint 2013.  It truly allows for a load balanced, highly available SharePoint farm where a user no longer needs to retain server affinity.  If I logged in on web front end 1 somehow get routed to web front end 2, no issues, lets just pull your login tokens, viewstate, and news feeds from this new cache shared across all the servers.

That said, so far maintaining it in a large enterprise environment has been a challenge. I have the opportunity(curse) of working in an environment of hundreds of SharePoint servers on premises. Our 2013 footprint is quickly growing; quickly enough that we can’t keep on top of say, manually, gracefully shutting down our distributed cache services every time an OS patch window rolls around.

Oh and what about those times that Windows just doesn’t want to play nice and a server blue screens, or worse just hangs. At this point we’re left with a server with an empty cache, which every other server in the farm thinks should have all of our logon tokens!

SP: Access Denied!!!
Browser: but I have a valid FedAuth cookie still!
SP: Yeah well I have no idea what happened to your claims so I’m not sharing
Browser: but…FedAuth
SP: I’m just trying to be nice, but I just don’t think this site has been shared with you.

So where do we go from here. The ideal situation is that before a cache server is restarted, the service is gracefully stopped, then removed.

Stop-SPDistributedCacheServiceInstance -Graceful
Remove-SPDistributedCacheServiceInstance
*thank the MS Gods for autocomplete…

And after reboot, re-added.

Add-SPDistributedCacheServiceInstance

But, can we fire that automatically for all reboot situations? Well, no, not easily.

1) Any shutdown tasks will not save you in the case of a blue screen, hardware failure, or a guy in the data center tripping over the power cord.   The other servers in the farm will not know to rebuild the cache, it’ll just keep sending the requests to the bads server.

2) There is no ideal way to run the Powershell on start up, as a farm account, and please don’t tell me your farm account is Local Service or Network Service on a multi-server farm.  The closes you can come is a start up script assigned to the local policy that calls a secondary script that has hardcoded credentials to impersonate a farm admin account.

What we’re doing:

1) record the last time each server had its distributed cache gracefully started.  SPServer.Properties is a good place for that

2) Every 30 minutes or so, check this against the server’s last boot time (WMI)

3) If the last reboot is more recent than the last graceful dist cache start up, tell the farm to rebuild that server’s cache.

Sprinkle in some (lots of) error handling and call it a day.

Things that would compel me to buy a smartwatch

Archive, Personal

1) self contained data connectivity that doesn’t cost me more than my kids college fund each month. You would think that at one point wireless bandwidth would be great enough that networks would no longer need to roll out new towers/technology and it would all become just paying for wear and tear of the network. It would become a utility and would just need maintenance and upkeep.

2) bioelectric charging – If I’m wearing it all day, why can’t it feed off my ridiculous metabolism. I’ll admit I have no idea if the human body could generate enough energy to power a device regardless of chip advances but wouldn’t it be awesome to never have to charge the thing.

3) Control for everything – the Internet of things is already in place and my watch is my easy button.

That said, I’d probably end up buying one much sooner than any of these are feasible, but sometimes it’s nice to think of the end game and not just the next step.

SharePoint 2013 App Domain dedicated Port binding vs dedicated IP binding

Archive, SharePoint, ,

I’ve recently been researching into what exactly is needed for implementing the App Model in SharePoint 2013.  Notably the environment into which we would be deploying SharePoint 2013 doesn’t support creating multiple IPs on a single box, so I had to get creative with the IIS bindings and instead use port separation.

The issue with this is that this isn’t a documented use case by Microsoft, so time to dig under the covers.

The app domains that you configure for a farm are essentially root domains for automatically generated hostname based site collections (HNSC).  You give SharePoint an app domain of myspapps.com and SharePoint dynamically generates HNSCs with each app “install”.

This is why the Site Subscription service (which powers HNSCs) is required to enable apps.  So we set up a wildcard DNS entry, point it at some sort of reverse proxy or load balancer, and direct it at our servers on a custom nonstandard http/https port.

It would be great if this worked out of the box, but for any redirect URLs that get sent back to the browser client, we see that the custom port number is sent back.  If you don’t have your proxy or load balancer configured to take this into account, you end up with 404s.  This is most evident in the out of the box defaultcss.ashx.  Your app will load but the out of the box stylesheets will fail.

In normal site collections/web applications this is not an issue as we can set up alternate access mappings.  Public = reverse proxy URL, internal = URL on custom port, or in the case of an HNSC, the commandlet you use to create the site collection also allows you to specify the port.

Microsoft already recognized a limitation in that apps could not support multiple auth zones, or HNSCs and provided a solution in the March PU They added a few new commandlets for generating per web-application app domains with support for multiple zones as well.  The new commandlets take advantage of more of the HNSC functionality and as a side effect it now allows for specifying a port for an app domain mapping.  While the documentation covers configuration at a per zone/HNSC app domain, it does go far enough to state that a single app domain can be shared so long as the authentication scheme and app pools match.

This gives us the ability to have the same exact functionality with the ability for SharePoint to properly resolve and regenerate URLs with the proper public URL format regardless of the IIS binding. Follow the link at the source for instructions on how to leverage the new commandlets if you’re running into challenges enabling Apps with a separate listener URL and public URL. You’ll want to pass the app domain and port for the public URL along to the commandlet and make sure your reverse proxy/load balancer passes on the Host: header to match.

Commandlet: New-SPWebApplicationAppDomain
Source: http://technet.microsoft.com/en-us/library/dn144963.aspx

*Note, the new commandlets are not yet in the commandlet reference since those haven’t been published since July 2012.
http://technet.microsoft.com/en-us/library/ee890108.aspx

What a geek does on vacation

Archive, Personal, SharePoint

How does a computer geek spend his Christmas vacation?
a) Moving his WordPress blog to his own hosted domain which has been sitting idle for well over a year?
b) Firing VirtualBox back up and getting multiple copies of Linux, Android, OS X, Windows Server 2012, and Chrome OS running on a single machine?
c) Installing a standalone SharePoint 2013 instance?
d) All of the above?
Sadly, its d).

I’m a bit of a nut with trying out OSs, I’ve had VMs spread out amongst my gaming rig and laptop, and a little jonesing for some OS X love from the old third Gen iBook I had (yes white ibook, not macbook). So what better to do in my spare time away from work than to take inventory of all my VMs and get them up and running on the big rig now that the laptop has been sacrificed for a clean Windows 8 install?

Inventory:

Laptop: Windows 8 Pro x64
PC: Windows 7 x64 (Virtual Box host for the following) (4×8 host machine, 3.2Ghz AMD Phenom II X4 955 BE)
VM: Windows Server 2012 Standard x64 (SharePoint standalone farm – didn’t feel like installing full SQL yet…)
VM: Ubuntu 11.10 – Sandbox Ubuntu box – upgrading to 12.04 and likely on to 12.10.
VM: Ubuntu 10.04 – Hosting some old php projects that I need to migrate somewhere else more robust
VM: Android x86 – Jelly Bean tablet mode – haven’t tried this image yet
VM: AndroVM – Jelly Bean the easy way. Download, unzip, import into VirtualBox
VM: ChromeOS – ChromeOS machine that I haven’t touched in years, not even sure if it works (Snow Leopard since I’m running AMD hardware)
VM: OSx86 – I honestly don’t remember the building this but it booted to a fresh OS X Snow Leopard install

To Do:
Move the site off the old Ubuntu VM to the cloud and retire the machine
Pick which Android VM is easier to work with and retire the other
Verify/Update ChromeOS
Play around with SP 2013 if the win2k12 VM ever starts performing better…might be the guest extensions (video driver)
Build some additional browser testing machines – IE6/IE7/IE8/XP/Vista ?
Suggestions (besides getting a life and more RAM…)?

Ghosts 2007->2010 Upgrade

Archive, SharePoint,

Ghosts can cause a massive headache when upgrading from SharePoint 2007 to SharePoint 2010, especially if you decide that the upgrade is a good time to re-organize your feature folder hierarchy.

On a recent upgrade, we ran into multiple issues related to ghosted locations for both page layouts and content types.

Page Layouts

When upgrading the solutions to 2010, the code was migrated into clean Visual Studio projects.  What this meant was that all the feature folders were now “cleaned up” by VS, instead of Features/featurename we now have Features/projectname_featurename.  This blew up all of our masterpages and page layouts.  We would get file exceptions from them regardless if they were ghosted or not.  The only solution is either to rename the feature folders and get the file references which are read-only in the database back to a happy state, or rip out all the old masterpage gallery items and recreate them with the new vti_setuppaths.  Given the size of our site, setting all the pages to a dummy layout while we did this wasn’t feasible so we had to rename all the folder to get our ghosts back.

So in the end masterpage gallery = keep your ghosts in the same place, or risk breaking your content.

Rename feature folders in VStudio 2010:
http://blogit.create.pt/blogs/andrevala/archive/2010/08/21/Renaming-a-Feature-in-SharePoint-Tools-for-Visual-Studio-2010.aspx

Content Types

These were another big headache.  To make the content type definitions more manageable, each content type was broken out into its own xml module, and all the fields that they reference were put into one module as well.  The same feature name is now in place, but the xml definition files went from one xml to about 15.

When the database was upgraded and new solutions were deployed we kept getting missing columns and content types.  The features wouldn’t even activate without a -Force flag.

We ran into the same exact situation as mentioned in the below link where the Definitions in the database were null, and upon upgrading, SharePoint didn’t know where to locate the actual xml that defined these fields and content types.

http://social.msdn.microsoft.com/Forums/en/sharepoint2010setup/thread/12effe76-af9d-424a-ab05-6f87d794ded9

There have been other, better posts about ContentTypes made by others but long and short of it is, modifying ContentTypes and fields that are feature deployed at the xml definition = bad.  In 2007 the changes would not propogate to child types properly.  In 2010, the overwrite flag helps, but mileage varies depending on if the ContentType is unghosted vs ghosted.

Looking into the suggested fix at the bottom of that post (modify the site columns in 2007) lead me to believe that these null missing items are coming across in the situations where the feature defined items were ghosted.  If there is a definition on the filesystem, why eat up the database storage by pushing it in there at the start.  We’ll only add the value once the item becomes unghosted.

We’re testing this out now, but I’m comfortable saying that with Content Types
 ghosts = bad; get rid of them.

So, if you’re upgrading and making structure changes to your feature folders and don’t want to modify any of the content, make sure to keep your masterpage gallery items in the same place reghosted to pickup changes and unghost your contenttypes so that they don’t blow up when their xml references are different.