Amy Jeanroy rightly submits the terminology "ornamental herbs" to a bit of scrutiny. After all, depending upon whom you ask, just about any herb could be considered "ornamental" if the term is based solely on appearance. So why do we even bother speaking of "ornamental herbs?"
Amy places this usage in a historical context. "Traditionally, ornamental herbs would get their names because although they sounded slightly familiar, they were not edible," explains About.com's Guide to Herb Gardens.
Of course, some ornamental herbs double as plants that have medicinal and/or culinary uses; 3 of the examples Amy cites are the following ornamental herbs:
Related resource: Landscaping With Herbs