In an oft-quoted line, George C. Scott claimed that what separated good actors from great ones was a quality he called, "joy of performing." He singled out James Cagney as an example and anyone who has seen Cagney hoof it in Yankee Doodle Dandy knows exactly what he meant.
When I look at a performance, I always keep Scott's wise words in mind, and I look for some other things besides. What is the commitment level of the actor? In other words does the actor hedge or in any way pull back from the more difficult or uncomfortable aspects of the role (the actor should not!)? I also ask what does the role require at the most basic level of competence and does the actor go beyond that. Along with George C. Scott's Joy of Performing, I think the ability to exceed the requirements of a role kicks things over from good to great.
As evidence I present my two favorite acting performances this year to date: Amy Adams in Enchanted and Emile Hirsch in Into the Wild.
In very different roles in very different movies, both show total commitment, unfettered joy, and the ability to see beyond the bare bones of a character to create fully fleshed human beings on screen.
Adams's accomplishment in this area may be the starker because her character as written is literally two dimensional: an intentionally cliched Disney princess come to life. But Hirsch does the same kind of work. Think how easy it would have been for Christopher McCandless to come off as tragic or sad or perpetually doomed. The story may be read as a tragedy, but Hirsch never, even in Chris's most desperate moments, never plays it that way.
Another litmus test for great acting is: does the performance lift the whole movie? Again, Amy Adams's work is a bit more visible (though no less impressive) because her movie aims a little lower. But if Emile Hirsch fails at any moment to embody his character, his movie collapses.
Great acting is hard to pin down precisely. These questions can help define some magnificent performances, but ultimately defining great acting is like the Supreme Court defining pornography (which I'm about to misquote):
"I can't say for sure what it is, but I know it when I see it."