Author Archives: donturn

About donturn

Don Turnbull, Ph.D. is a consultant specializing in software research and development focusing on search systems, information analytics, user experience design, semantic and knowledge management technologies as well as intellectual property analysis.

Quantitative Information Architecture recommended reading

Here is a brief list of recommended books from my Quantitative Information Architecturetalk at the 2010 Information Architecture Summit that review many aspects of quantitative thinking (both good and bad) that relate to using mathematical methods to as a toolkit for information architecture issues.

Quantitative Information Architecture Books

Many of these books are non-fiction favorites. I’ve used them in courses I’ve taught, relied on them for research ideas and used them to convey how quantitative innovation is pursued.

  1. The Control Revolution: Technological and Economic Origins of the Information Society by James Beniger. Nearly encyclopedic in its coverage of the Industrial Revolution’s impact on creating the Information Age, where economic forces accelerated collecting, storing and capitalizing on data. Particularly interesting (truly!) are insights about the railroad industry and information technology (e.g the telegraph).
  2. Against the Gods: The Remarkable Story of Risk by Peter L. Bernstein. Just thinking about this book makes me want to read it again. It’s a swashbuckler of a story of the history of people using mathematics to tame the world. (Well, at least to me.) Bernstein’s style is surprisingly readable with narratives that keep you engaged.
  3. Excel Scientific and Engineering Cookbook by David M Bourg. A great (but aging) overview of doing statistics in spreadsheets, including regression and time series analysis. Not for beginners, but a good reference and reminder of the power of Excel for almost all manner of analysis. (The only downside to Excel is its limit for working with very large datasets.)
  4. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century by David Salsburg. Another fun read, a glance through the history of some of the more famous statisticians (my favorite being Andrey Nikolaevich Kolmogorov and a partial history of Soviet science).
  5. Super Crunchers: Why Thinking-By-Numbers is the New Way To Be Smart by Ian Ayres. The most readable (and current), with some basic introductory ideas presented in the context of how organizations such as Netflix, Southwest Airlines – and of course Google use numbers and industries including baseball and wine-making are impacted by quantitative work.
  6. The Rise of Statistical Thinking, 1820-1900 by Theodore M. Porter. This book is mostly thematic, covering the rise of statistics and their influence in the social sciences. A bit dry (and poorly typeset) but a foundational study. (Feel free to rely on the Index to jump around to people or topics you might be more interested in.)
  7. When Information Came of Age: Technologies of Knowledge in the Age of Reason and Revolution, 1700-1850 by Daniel R. Headrick. This book was a quick read, suggesting a number of common themes such as the rise of the Age of Reason and the parallel development of scientific instrumentation. As empirical sciences progressed, a resulting increase in collected data brought forth the origins, expansion and professionalization of many kinds of information systems including graphs, maps, encyclopedias, the post office and insights of key scientists of the age (e.g. Carl Linnaeus). Not as grand in scope as other recommended books, but focuses more clearly on types of information that are often the focus of IA efforts.
  8. Men of Mathematics by E.T. Bell. A somewhat stilted (written in the 1930’s) biographical walk-through of many storied mathematicians (i.e the people’s names you hated to hear in 10th grade Geometry), that reveals the history of quantitative analysis and the intellectual vigor (did I just say that?) of those like Gauss or Lagrange. Even if the math itself is not your normal interest, this book is an index of obsession, diligence and ingenuity.
  9. The History of Statistics: The Measurement of Uncertainty before 1900 by Stephen M. Stigler (not shown). I have not finished this book, and there is a lot in it that I do not have much interest in, and have put it down several times (it is a bit dry). However, the integration of how different statistical measures were built progressively is interesting. Also, one of the better sets of discussion about Karl Pearson.

Quantitative Information Architecture Books

    Two books illustrate the downfall of quantitative hubris (among other things) and both are fun to read.

  1. When Genius Failed: The Rise and Fall of Long-Term Capital Management by Roger Lowenstein. This book narrates the catastrophic failure of Long-Term Capital Management, the fabled sure-bet genius-powered hedge fund that boasted two Nobel laureates among its partners and how they nearly crashed the entire world financial system with this overconfidence in 1998.
  2. Too Big to Fail: The Inside Story of How Wall Street and Washington Fought to Save the Financial System—and Themselves by Andrew Ross Sorkin. A detailed (600 page plus) report of the nearly minute to minute recent financial crisis and an indictment of over-reliance on trusting abstract mathematics without (any?) explanation or validation. Worth remembering when confronted with abundant or seemingly infallible data-driven results that we should not be intimidated and remember to ask Why? and How?.

Personalized Search

Personalized Search: A Contextual Computing Approach May Prove a Breakthrough in Personalized Search Efficiency


James Pitkow, Hinrich Schuetze, Todd A. Cass, Rob Cooley, Don Turnbull, Andy Edmonds, Eytan Adar, et al.


A contextual computing approach may prove a breakthrough in personalized search efficiency.


Contextual computing refers to the enhancement of a user’s interactions by understanding the user, the context, and the applications and information being used, typically across a wide set of user goals. Contextual computing is not just about modeling user preferences and behavior or embedding computation everywhere, it’s about actively adapting the computational environment – for each and every user – at each point of computation. (p 50)

The Outride system was designed to be a generalized architecture for the personalization of search across a variety of information ecologies.(p 52)

Search Engine - Average Task Completion Time in Seconds

While the results may seem overwhelmingly in favor of Outride, there are some issues to interpret. First, some of the scenarios contained tasks directly supported by the functionality provided by the Outride system, creating an advantage against the other search engines. Indeed, Outride features are specifically designed to understand users, provide support by the conceptual model and tasks users employ to search the Web, and to contextualize the application of search. This is the goal of contextual computing and why personalizing search makes sense.

Second, while the use of default profiles could have provided an advantage for Outride, it also could have negatively influenced the outcome, as the profile did not represent the test participants’ actual surfing pat- terns, nor were the participants intimately familiar with the content of the profiles. Third, some of the gains are likely due to the user interface since the Outride sidebar remains visible to users across all interac- tions, helping to preserve context and provide quick access to core search features. For example, while search engines require users to navigate back and forth between the list of search results and specific Web pages, Outride preserves context by keeping the search results open in the sidebar of the Web browser, making the contents of each search result accessible to the user with a single click. Still, the magnitude of the difference between the Outride system and the other engines is compelling, especially given that most search engines are less than 10% better than one another. (p 54)


information retrieval, search, information seeking, relevance feedback, personalization, contextual computing, user interfaces, search process

Cite As

Pitkow, J., Schutze, H., Cass, T., Cooley, R., Turnbull, D., Edmonds, A., et al. (2002). Personalized Search: A Contextual Computing Approach May Prove a Breakthrough in Personalized Search Efficiency. Communications of the ACM, 45(9), 50-55.

References in this publication

  • Anderson, J.R. Cognitive Psychology and Its Implications. Freeman, San Francisco, CA, 1980.
  • eTesting Labs. Google Web Search Engine Evaluation;
  • Pirolli, P. and Card, S.K. Psychological Review 106, 4 (1999), 643–675.
  • Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986

Publications that cite this publication

Advertising Academia With Sponsored Search: an exploratory study examining the effectiveness of Google AdWords at the local and global level


Don Turnbull and Laura F. Bright


An exploratory study conducted in late autumn and early winter 2006-2007 investigates the purchasing of sponsored search advertising for a major US university’s academic department. The ad campaign used Google’s AdWord service with the goal of increasing awareness of the academic department and encouraging potential graduate admissions or admissions inquiries. A behavioral model of information seeking is suggested that could be applied for selecting appropriate types of online advertising for awareness and other advertising goals. The study found little overlap between traditional, commerce-oriented online advertising methods and a general awareness campaign, as evidenced by a low click-through rate to the targeted site. Insights for future studies include increased integration with server logs, targeted site query terms and alternative awareness strategies.


sponsored search; online advertising; search engines; behavioral model; information seeking; electronic business; Google.

Cite As

Turnbull, D. and Bright, L.F. (2008) Advertising academia with sponsored search: an exploratory study examining the effectiveness of Google AdWords at the local and global level, Int. J. Electronic Business, Vol. 6, No. 2, pp.149-171.

References in this publication

  • Ad Age Search Marketing Fact Pack (2006) Published by eMarketer on 6th November, Retrieved online on 01/19/07.
  • Allen, T.J. (1977) Information needs and uses, Annual Review of Information Science and Technology, Vol. 4, pp.3-29.
  • Chang, S. and Rice, R. (1993) Browsing: a multidimensional framework, Annual Review of Information Science and Technology, Vol. 23, p.242.
  • Cho, C. (2003) Factors influencing the clicking of banner ads on the World Wide Web, Cyberpsychology and Behavior, Vol. 6, No. 2, pp.201-215.
  • Cho, C–H. and Cheon, H.J. (2004) Why do people avoid advertising on the internet?, Journal of Advertising, Vol. 33, No. 4, pp.89-97.
  • Choo, C.W., Detlor, B. and Turnbull, D. (1998) A behavioral model of information seeking on the web – preliminary results of a study of how managers and IT specialists use the web, Proceedings of the 61st Annual Meeting of the American Society of Information Science, Published for the American Society for Information Science by Information Today Inc., Pittsburgh, PA.
  • Choo, C.W., Detlor, B. and Turnbull, D. (2000a) Web Work: Information Seeking and Knowledge Work on the World Wide Web, Kluwer Academic Publishers, Dordrecht, The Netherlands.

  • Choo, C.W., Detlor, B. and Turnbull, D. (2000b) Working the web: an empirical model of web use, 33rd Hawaii International Conference on System Science (HICSS), Maui, HI.
  • Coulter, R.A., Zaltman, G. and Coulter, K.S. (2001) Interpreting consumer perceptions of advertising: an application of the Zaltman metaphor elicitation technique, Journal of Advertising, Vol. 30, No. 4, pp.1-21.
  • Coyle, J.R. and Thorson, E. (2001) The effects of progressive levels of interactivity and vividness in web marketing sites, Journal of Advertising, Vol. 30, No. 3, pp.277-289.
  • Ellis, D. and Cox, D. (1993) A comparison of the information seeking patterns of research scientists in an industrial environment, Journal of Documentation, Vol. 49, No. 4, pp.356-369.
  • Ellis, D. (1989) A behavioural approach to information retrieval systems design, Journal of Documentation, Vol. 45, No. 3, pp.171-212.
  • Ellis, D. (1997) Modelling the information seeking patterns of engineers and research scientists in an industrial environment, Journal of Documentation, Vol. 53, No. 4, pp.384-403.
  • Fain, D.C. and Pederson, J.O. (2006) Sponsored search: a brief history, American Society for Information Science and Technology Bulletin, Retrieved on 11/15/06 from, January, Special Issue.
  • Feng, J., Bhargava, H.K. and Pennock, D.M. (2005) Implementing sponsored search in web search engines: computational evaluation of alternative mechanisms, Journal of Computing, Vol. 19, No. 1, Winter 2007, pp.137-148.
  • Friestad, M. and Wright, P. (1994) The persuasion knowledge model: how people cope with persuasion attempts, Journal of Consumer Research, Vol. 21, pp.1-31.
  • Godin, S. (1999) Permission Marketing, Simon and Schuster, New York. Goral, T. (2003) Intelligent admission, University Business, Vol. 6, No. 3, pp.38-41. Holahan, C. (2006) Click Fraud: Google Comes Clean, Sort Of, BusinessWeek, 27 July. Jansen, B.J. (2007) Click fraud, IEEE Computer, Vol. 40, No. 7, pp.85-86.
  • Jansen, B.J. and Resnick, M. (2006) An examination of searchers perceptions of non-sponsored and sponsored links during e-commerce web searching, Journal of the American Society of Information Science and Technology, Vol. 57, pp.1949-1961.
  • Li, H., Edwards, S. and Lee, J-H. (2002) Measuring the intrusiveness of advertisements: scale development and validaton, Journal of Advertising, Vol. 31, No. 2, pp.37-47.
  • Marchionini, G. (1995) Information Seeking in Electronic Environments, Cambridge University Press, Cambridge.
  • Moore, R.S., Stammerjohan, C.A. and Coulter, R.A. (2005) Banner advertiser-website context congruity and color effects on attention and attitudes, Journal of Advertising, Vol. 34, No. 2, pp.71-84.
  • Shamdasani, P.N., Stanaland, A.J.S. and Tan, J. (2001) Location, location, location: insights for advertising placement on the web, Journal of Advertising Research, Vol. 41, No. 4, pp.7-21.
  • Stone, B. (2007) Dont like dancing cowboys? Results say you do, New York Times, Media and Advertising Section, 18th January.
  • Sutton, S.A. (1994) The role of attorney mental models of law in case relevance determinations: an exploratory analysis, Journal of the American Society of Information Science, Vol. 45, No. 3, pp.186-200.
  • Taylor, R.S. (1986) Value Added Processes in Information Systems, Ablex Publishing Corp., Norwood, NJ.
  • Xing, B. and Lin, Z. (2004) The impact of search engine optimization on online advertising market, ACM Conference Proceedings, Winter, pp.519-530.

Publications that cite this publication

Google Scholar Citations

Related Articles

Metropolitan Information Architecture at the 2010 Information Architecture Summit

My other presentation at the 2010 Information Architecture Summit in Phoenix this week is with the formidable John Tolva at IBM and is focused on city-scale information architectures, the data we swim through in urban settings and how designers can and should lead in shaping this information’s collection, use and display in the system that is a city.

Metropolitan Information Architecture – Don Turnbull and John Tolva

2:00 – 2:45PM on Sunday, April 11 in the Phoenix Room

If the future of the world is cities, how can we design user experiences at
city-sized scales? With digital interaction, are we all living in facets of the
same virtual city or does location still constrain us?

This panel will review and discuss recent research and some upcoming designs
that are only beginning to unveil how our interactions with both digital and
physical environments are changing including:

  • How does the actual architecture of information & design synchronize
    with urban architecture?
  • What city constraints including urban decay, congestion & energy
    consumption affect design and how can design improve them?
  • How does mobile communication and web culture impact the streetscape?
  • What can designers leverage from virtual worlds, augmented reality, MMO
    games and urban design?
  • Who are the people and cities that have embraced data/networks as
    matters of physical design (rather than value-add services to residents)?
  • Is geography fate? What does location mean for UX?
  • When does social media start to change digital & physical social spaces
    of the urban network?
  • What will metropolitan experiences be like in 10 years? 20?

The twitter hashtag for this talk is #metroia. Feel free to send me questions directly via twitter/donturn too.

Originally, I wanted to call this talk Cosmopolitan Information Architecture, inspired by Wynton Marsalis’ definition of cosmopolitan as meaning “you fit in wherever you go”, which should be a goal for anyone shaping experiences for living in a community.

Quantitative Information Architecture at the 2010 Information Architecture Summit

I am presenting on two different topics at the 2010 Information Architecture Summit in Phoenix this week.

The first talk is a set of ideas related to the work I’ve been doing recently, building data structures, crafting algorithms and designing user experiences that are powered by quantitative data.

Quantitative Information Architecture – Don Turnbull, Ph.D.

10:30 – 11:15AM on Saturday, April 10 in Ellis

Why quantitative information architecture? Why now?

You don’t have to be RainMan or Stephen Hawking to use numbers to get things
done. Quantitative methods are applicable for IA thinking be it for hypothesis
generation, instrumentation, data collection and analysis of information at
scales never before possible with insights that are comparable over time,
generalizable and extensible.

Quantitative skills can allow IAs to interpret and analyze others’ designs and
research more readily, as well as combine methods and models for meta-analysis
to help IAs move from description to prediction in designing and developing
future interfaces and architectures.

This presentation will review why you should use quantitative methods and
discuss both foundational and emerging ideas that are applicable for content
analysis, behavioral modeling, social media usage, informetrics and other
IA-related issues.

The twitter hashtag for this talk is #quantia. Feel free to send me questions directly via twitter/donturn too.

Quantiative Information Architecture slide deck from the 2010 IA Summit

My 2009 Austin Film Festival

I was very lucky this year to catch a lot of movies at the Austin Film Festival. I’m including parts of the descriptions for each movie in blockquotes with commentary of my own after each.

Matthew Weiner presents…Mad Men

Former Sopranos writer Matthew Wiener has created a masterful tale about one of New Yorks most prestigious ad agencies at the beginning of the 1960s. Focusing on one of the firms most talented and extremely mysterious executives Donald Draper. Weiner will be on hand for a special cocktail hour screening of one of his personal favorite episodes of the beloved TV show.

Weiner’s talk was the highlight of the festival for me. Why there was a TV-guy at the film festival I don’t know but I certainly enjoyed his talk about making Mad Men. We watched a recent episode: Guy Walks Into an Advertising Agency, which Weiner said was originally titled Nothing Runs Like a Deere. He said this episode was all about expectations and their anticipation. After watching it again with the audience, I saw many other ideas based on that theme. Most enjoyable was seeing the episode with a theater full of people. You forget how great the dialogue is, especially John Slattery as Roger Sterling. His timing is perfect and with a spirited, enthusiastic audience – had us roaring. Mad Men has been renewed for a fourth season too, so lots to look forward to.

American Cowslip

Ethan Inglebrink an eccentric agoraphobic heroin addict w ho is obsessed with his garden is searching for the perfect moment in a life that is running out of time. And the odds are against him. This darkly comedic tale set in small town California follows the last days of Ethan’s life as he struggles to find love and purpose – at a time when it might be too late to even matter.

This movie was interesting, especially for the cast of actors (Diane Ladd, Rip Torn, Cloris Leachman, Val Kilmer, Peter Falk, Bruce Dern). The plot is odd, and meanders a little in the middle (ok, a lot) but it was fun. I ran into the star and co-writer Ronnie Gene Blevins, the next night in line and he implied it was quite an experience. I commented that they should have featured the donkey more in the trailer, as a donkey always equals comedy gold.

Harmony And Me

The sharp-witted Byington returns with the Austin-shot story of Harmony a prototype nightmare dumpee – the sort of heartbroken friend you take out for consolatory drinks two or three times before resorting to any excuse to avoid hearing yet again the play-by-play rehash of what went wrong with his relationship. Never self-indulgent Byington creates a memorable modern comedy with subtle and sharp performance from his cast.

Locally made here in Austin and starring lots of local talent. Nice to see some Austin locations, but once you know Austin, you realize some strange liberties are taken with the driving about town. The audience was really behind this film, which made it fun to watch. Good dialogue and a few strong scenes, but I couldn’t tell if most of the movie was improvised or not. Not that it matters.

The Vicious Kind

A dramatic comedy set in small town Rhode Island brings tog ether elements of depression romance and self exploration among a dysfunctional family of men. Lonely and frustrated after a bitter break up Caleb warns his younger brother against women especially the pretty young thing he’s bringing home for the holidays. His indifferent opinion and steely resolve is put to the test as his attraction to Peter’s girlfriend begins to grow. The Vicious Kind features a very well written script and inspired performances by Simmons Scott and Frost.

Hard to classify, but interesting characters. I also didn’t know they had McMansions in Connecticut.

Houston We Have A Problem

With Houston We Have a Problem filmmaker Nicole Torre steps inside the world of Texas oil culture to understand just how the United States developed its large appetite for crude. Wildcatters independent oilmen and even the president of Shell Oil examine some hard truth s while discussing the history and the future of this limited commodity upon which we have become so dependent and weigh in on the future of this country’s energy supply.

A talky documentary about the history of oil, mostly in Texas, and what we’ve got to look forward to next. A surprising range of opinions (some unintentionally humorous). My big takeaway was to invest in energy stocks, which might not quite be the point.


In 1994 real-life crop artist Stan Herd traveled from Kansas to Manhattan’s Upper West Side to create a massive environmental artwork on land owned by Donald Trump. The multi-acre artwork was made from soil rock plants and vegetation near an underground railway tunnel. Stan recruited a number of homeless individuals living in the tunnel to become his crew. Over the months it took to complete the earthwork Stan dealt with a number of difficulties in bringing his unique rural art form to an urban canvas. In the process he unexpectedly encountered the true meaning of his art and it’s ultimate lasting rewards.

Nicely cast and acted movie. The most surprising thing about it was that the director, Chris Ordal, mentioned in the post-screening chat that it was shot on location in Kansas instead of somewhere in New York. Also, a supporting role by the great Zach Grenier.

Love the Beast

What if you were a Hollywood movie star with an obsession for cars and racing? You would probably read every script with even the tiniest link to the subject matter in hopes that you could tell a great car story the likes of Grand Prix Le Mans or Mad Max. Then one day you open your garage door and sitting there right in front of you is the inspiration that gets you thinking about that old saying If you want something done right Eric Bana has now unwillingly cast himself in his own real life drama and it is a love story between Eric and the car he has owned since he was fifteen. We follow him from inside the race car to the surreal world of the red carpet. The personal and social pressures mount up in the face of re-building a car that means so much to him.

Finely directed and edited movie. This surely had a larger budget behind it over most at the festival and it showed. Nice driving cinematography and some candid moments for Bana and his group of friends.


A world that’s more riot than profession, the trading floors of Chicago are a place where gambling your family’s mortgage is all in a day’s work. Floored, at a time when markets are unhinged, offers a unique window to this lesser-known world of finance. These men may not have degrees, but they’ve got guts, and penchant for excess that solicits simultaneous feelings of revulsion and admiration. But like many aspects of our economy, technology is changing the way these traders do business, and these eccentric pit denizens aren’t the type to take kindly to new tricks.

An interesting insight into some computer-phobic people. Very candid (and amusing) commentary by several interviewees.

The Donner Party

During the winter of 1846, a group of westward bound settler s is stranded in the Sierra Nevada Mountains. They must reach California for salvation, but without food and a guide, the group becomes disillusioned and decisions must be made so that others can stay alive. Inspired by the infamous Donner Party tragedy, the film is a harrowing look at survival and how far some will go to achieve it.

This movie looked beautiful, and the performances were on target. I was reminded a lot of Deadwood. However, I felt that with the casting (Glover) I was expecting something a bit more divergent (quirkier). I left this movie about half-way through because of the disrespectful behavior of others in the audience. People on both sides of me had their phones out throughout the movie and the festival staff let others in late with some clamor. One late-arriving couple sat near me and chatted/mumbled amongst themselves continuously.

Up in the Air

The timely odyssey of Ryan Bingham, a corporate downsizer and consummate modern business traveler who, after years of staying happily airborne, suddenly finds himself ready to make a real connection. Ryan has long been contented with his unencumbered lifestyle lived out across America in airports, hotels and rental cars. He can carry all he needs in one wheel-away case; he’s a pampered, elite member of every travel loyalty program in existence; and he’s close to attaining his lifetime goal of 10 million frequent flier miles and yet . . . Ryan has nothing real to hold onto. When he falls for a simpatico fellow traveler, Ryan’s boss, inspired by a young, upstart efficiency expert, threatens to permanently call him in from the road. Faced with the prospect, at once terrifying and exhilarating, of being grounded, Ryan begins to contemplate what it might actually mean to have a home.

In some ways it is unfair to compare this movie to the others. Obviously working with a much larger budget gives you a different kind of film. I’ll summarize that I liked it and remember hearing the book is interesting too. Reitman did a post-screening Q&A and was highly entertaining and talked a lot about his process writing (and re-writing) and directing.

pre-SXSWi meetup Thurs March 12th, 5-7pm at The Cedar Door

As we’ve done in the past, let’s all get together for a pre-SXSW meetup Thurs March 12th, 5-7pm at The Cedar Door, 201 Brazos St. (2 blocks from the Austin Convention Center).

If you’re getting to Austin for SXSW Interactive just a little early, come on by and get your SXSW started with some others in town a day early too, not to mention meet some of us lucky enough to live in Austin. Feel free to forward this to others you know coming to Austin for SXSWi too.

Those of you in Austin, tell your Web/IA/Designer/Startup/SWSXi-like friends to meet us there. (Look for the geekiest crowd at the Cedar Door, trust me, that’ll be us. Probably on the East patio.)

Here’s an upcoming link for the event, if you’re into that kind of RSVPing.

Note that now you can pick your SXSW interactive badge on at the Austin Convention Center the same evening, come to the meetup, then pick up your badge!