Meta-tags for mobile better optimization

The simple answer is: viewport is good, the others are... less good.

viewport

viewport is a widely supported de-facto standard - originally created by Apple for mobile Safari on iPhone, it's been adopted by almost all other mobile browsers: Opera Mobile, iPhone, Android, Iris, IE, BlackBerry, Obigo, Firefox

Simple example use case: make the site full width on mobile:

<meta name="viewport" content="width=device-width, initial-scale=1.0"/> 

The other two are older de-facto 'standards' for 'feature phones' - which are generally too old to support viewport:

HandheldFriendly

This tag was originally used to identify mobile content in AvantGo browsers, but became a general standard for identifying mobile websites. However, it’s unknown what range of browsers support this meta tag:

<meta name="HandheldFriendly" content="true"/> 

MobileOptimized

This is a Windows-proprietary meta tag that also eventually became used as another means of identifying mobile content. The drawback with this tag is that a specific width must be given. Again, it’s unknown what the support for this tag is:

<meta name="MobileOptimized" content="320"/> 

Summary

Use viewport unless you need to support older feature phones which don't support it, in which case, probably use both HandheldFriendly & MobileOptimized - but test your target devices and find out.

Should they be used? Where and when is it appropriate to use them? Are there alternatives (without user-agent recognition)?

They should be used when you want the effects they create - generally, telling phones what default zoom to use, controlling re-sizing, etc. This is a good explanation of why you might want to use viewport, for example: http://davidbcalhoun.com/2010/viewport-metatag - it also lists the other properties that you can set with viewport and what they do.

They only other way to achieve these effects, without using these metatags, is with funky JS tricks - which will be slower, require script loading, be difficult to maintain and will be unreliable. Browsers that don't support viewport will probably have very buggy JS interface to viewport related stuff; see the quirksmode links below.