"They Can't Take That Away from Me": History, Commentary, Discography, Performances on Video

"They Can't Take That Away from Me"

As virtually everybody in the movie believes the delicious rumor that Fred (Pete Peters, aka Petrov the great Russian ballet dancer) and Ginger (Linda Keene, night club performer) are married, they come up with the screwball idea to tie the knot so they can then immediately get divorced, thereby putting the rumors to an end. They cross the Hudson from Manhattan to New Jersey where the clerk at the marriage license bureau assures them that the grounds for divorce in his state are just one thing: "marriage." Taking the ferry back to New York, Ginger laments she never knew "getting married was so depressing." It's depressing, of course, because she does love Pete (and he her) and the next day they plan to get divorced. Fred's deeply touching response is that even though the divorce will take away their marriage much else of great value will remain, as he sings, "They Can't Take That Away from Me." (See video clip below). Part 3 of "The Music of Shall We Dance" contains relevant commentary.
(Details about movie at IMDB)

Video Notes: Fred Astaire sings to Ginger Rogers in the RKO movie Shall We Dance. This is the song's introduction and includes the verse (with a word or so chopped off right at the beginning). The video is the same as the album track but also adds on the finale of the movie with Fred and Ginger reprising "They All Laughed," letting us know they didn't take her away from him after all.

1) Flying Down to Rio (1933); 2) The Gay Divorcee (1934); 3) Roberta (1935); 4) Top Hat (1935); 5) Follow the Fleet (1936); 6) Swing time (1936); 7) Shall We Dance (1937); (8) Carefree (1938); (9) The Story of Vernon and Irene Castle (1939); 10) The Barkleys of Broadway (1949).

William Zinsser points out that between 1935 and 1938 "five of America's best songwriters wrote 28 songs that collectively stand as a museum exhibit of the form," a form, which for our purposes, are prototypical for songs in The Great American Songbook. This happened, he claims, "because of the influence of one man"--Fred Astaire. The songwriters he was referring to are Irving Berlin, Jerome Kern, Dorothy Fields, George Gershwin and Ira Gershwin, all of whom traveled to Hollywood from New York to write songs for Astaire movies. The movies were Top Hat (1935) and Follow the Fleet (1936) with songs by Berlin, Swing Time (1936) with songs by Kern and Fields, and Shall We Dance (1937), A Damsel in Distress (1937) and Carefree (1938) with songs by the Gershwins. (Astaire's co-star in all but Damsel in Distress was Ginger Rogers).

Ed's note: The video just below contains commentary about "They Can't Take That Away from Me" but begins with comments on another song from the movie, "They All Laughed," and George's famous background music for Fred's and Ginger's "walking the dogs" scene.

(Part 1 and Part 2 of this documentary can be viewed on our page
for the song "Let's Call the Whole Thing Off.")
(Please complete or pause one
video before starting another.)

The Gershwins flew from New York (actually Newark Airport) to Los Angeles on August 8, 1936, having signed a sixteen week contract with RKO Radio Pictures to work on an unscripted film, a film not yet bearing the title Shall We Dance. They had only a vague summary and the knowledge that it would star Fred Astaire and Ginger Rodgers. The brothers had already written a couple of songs at least partly in New York. They managed this based on the preliminary material provided them by the studio and their knowledge of the kinds of numbers they had seen in previous Astaire-Rogers films. Even after they arrived in Hollywood there were many delays before they could even get a final script.

The closest thing to an exact date for the composition of "They Can't Take That Away from Me," comes from a letter of George's dated October 24, 1936, (just under three months after their arrival in California) in which he writes that we are "on the last lap of our score." He continues, "Ira and I have written a song called 'They Can't Take That Away from Me" which I think has distinct potentialities of going places . . . " (Jablonski and Stewart, p. 246, hard cover Ed.). The song turned out to be one of George's favorites and one that had great significance for Ira after the untimely death of his brother from a brain tumor less than a year after the song was completed. (See Michael Feinstein's commentary below)

George was not happy with the way the studio dealt with "They Can't Take That Away from Me." From the time he was fifteen years old working as a song plugger (one whose job it was to play and/or sing songs from a music publisher's catalog in order to pitch them to performers and other prospective buyers) on Tin Pan Alley, George Gershwin believed that a song could not do well without being given proper exposure, without being plugged. As late as this trip to to the West Coast he felt the same way and complained that the "Hollywood filmmakers had literally thrown 'one or two songs [they had written for Shall We Dance] away without any kind of plug.'" Most upsetting to him was what happened to "They Can't Take That Away from Me." In the final cut of the movie, Astaire sings the chorus through only once with the last refrain dropped altogether and does not dance to it at all. Gershwin felt RKO had relegated the song to a less than featured spot and that, unlike on Broadway, there was nothing much he could do about it.(Jablonski, p. 41).

Fred sings it to Ginger outside a shadowy Manhattan ferry terminal after their return from a quickie New Jersey marriage undertaken only so they can get a quickie divorce. Portions of the melody are used for background music in a couple of other spots in the film, but other than that -- nothing (See Video Playlist, #1). George was partially placated by the recording, which he quite liked, that Astaire made of the song with the Johnny Green Orchestra shortly after the film's release as well as by the comments about "They Can't Take That Away from Me" made by Irving Berlin and Jerome Kern, who called it a "lasting song" and (as George wrote in a letter) "one of the best songs Ira and I have written in a long time" (Pollack, p. 675).

George Gershwin and Fred Astaire first met in NYC when both were teenagers. George was a songplugger on Tin Pan Alley and Fred and his sister Adele had come in looking for material. George commented to Fred that it would really be great if some day I wrote a show and you starred in it. Reportedly the last thing George uttered before he died was Fred Astaire's name. )

"You Can't Take That Away from Me" came into being when "George had an idea for a melody, a simple but haunting rhythmic manipulation of a single tone" beginning with four repeated notes -- three quarters and an eighth. After listening, Ira, his brother and lyricist, responded by saying, "If you can give me two more notes in the first part, I can get, 'The way you wear your hat'," which of course became the beloved first phrase of the first refrain -- the first thing they wouldn't be able to take away from him (him being Fred Astaire's character) because they were getting divorced the next day. After that it didn't take the brothers long to finish the song, and George was soon playing it -- plugging it -- at the houses of his recently transplanted New York friends, including Harold and Anya Arlen. "He Loved it," Arlen remembered, "and every time he played it he asked Annie -- in her little voice -- to sing" (Jablonski, p. 302).

Besides "the way you wear your hat," another item on Ira's list of what they couldn't take away was "the way you sing off key." Ira had cleverly managed to include a negative trait but in a way that made the listener understand it not as negative at all but as charming and appealing. Not long after Shall We Dance was released, George and his friend S. N. Behrman (the American playwright and screenwriter) were riding through Beverly Hills discussing how much they admired Ira's work, when Behrman cited this particular line to George saying "how marvelous it was of Ira to have added singing off-key to the list of the heroin's perfections -- how it bathed nostalgia in humor." (Rosenberg, p. 342). They must have concurred on how well the line's playful irony manifested Ira's genius.

In general the film's songs were not received as enthusiastically as were those from previous Astaire-Rogers movies. "Let's Call the Whole Thing Off" and "They Can't Take That Away from Me" did make The Lucky Strike Hit Parade, the Fred Astaire version (with Johnny Green mentioned above) rising to number one in April, 1937. Later that year, Ira said that "he and George were 'pretty proud' of a score they considered 'smart' and ' a little sophisticated,' but wondered, 'Maybe that was a mistake to put so many smart songs in one picture"
(Pollack, p. 677).

George lived long enough to know about the song's success on the The Hit Parade despite the film's luke warm reception but not long enough to know of its nomination for an Academy Award. It did not, alas, win. The Academy, its inscrutable choices apparently extending back to near its very beginnings, gave the 1937 prize to the song "Sweet Leilani" by Harry Owens sung by Bing Crosby in the movie Waikiki Wedding. The ever acerbic Oscar Levant, a very close friend of the Gershwins, reportedly responded to the Academy's selection by commenting, "I'd like to say something about Harry Owens. His music is dead, but he lives on forever (Pollack, p. 677). To cap off the absurdity of the moment, the Oscar was presented by Irving Berlin.

Years later, in his book Lyrics on Several Occasions, Ira makes the following observation (confirming his wry attitude toward life) about his encounters with the Academy of Motion Picture Arts and Sciences:

They Can't Take That Away from Me" was a Best Song nominee in the . . . "Oscar" awards. And since then, if I check correctly, only two others of mine have been nominated: Long Ago and Far Away" and "The Man That Got Away." These three songs have two things in common: (1) in the final voting none won; and (2) the title of each contains the word "away." So? so -- away with "away"? ( Ira Gershwin , Lyrics , p. 48, soft cover Ed.)

Michael Feinstein (who, as a very young man, was an assistant to Ira Gershwin for six years collecting and annotating the Gershwin archive for the Library of Congress) recollects in his autobiography the personal significance to Ira of the songs he and George wrote for Shall We Dance. Feinstein notes that after George died, Ira was very depressed, essentially non-functional with regard to work:

Harold Arlen and Jerome Kern tried to get him to write by sending him home recordings of melodies (which I [Feinstein] later came across in Ira's closet). One day in a stupor he stumbled over to the phonograph and put on one of the records that Johnny Green had recently made with his dance band and Fred Astaire* of the songs from the movie Shall We Dance . They included . . . " Let's Call the Whole Thing Off " and "They Can't Take That Away from Me." As Ira listened to the music, the vitality of the songs and the performances went to his heart; that was the first time he started even in a small way to feel better. Hearing those recordings was the thing that most helped Ira to assuage his grief ( Feinstein, pp. 74-75) .

The Barkleys of Broadway, produced at MGM in 1949, was the last movie Fred Astaire and Ginger Rogers made together, twelve years after Shall We Dance and eleven years after George's death. It had a score with music by Harry Warren and lyrics by Ira Gershwin but also included one wholly Gershwin song, "They Can't Take That Away from Me" sung by Fred and danced by Fred and Ginger. It was almost "as if 'They Can't Take That Away from Me' finally got the plug George had thought it merited but did not get in Shall We Dance (Jablonski, p. 354).

The songwriter Hugh Martin referring to The Barkleys of Broadway version of the song, notes, somewhat wistfully, that although the song's later incarnation in the MGM production uses a grander arrangement, is accompanied by a bigger orchestra and includes a Fred and Ginger dance routine, it is, for his money, less effective and not as moving as the more intimate RKO version in Shall We Dance (Commentary by Hugh Martin and Kevin Cole, Astaire & Rogers Ultimate Collector's Edition/Shall We Dance disk).

Alec Wilder in his classic study, American Popular Song is not crazy about the songs from Shall We Dance, with the exception of "They Can't Take That Away from Me," which he not only says is "the best" but follows up with, "It's etched like acid into my memory bank."

Wilder cites several qualities that are particularly appealing to him: its "extremely inventive" verse which "gives not a hint of what is to follow"; George's repeated notes, a fairly common Gershwin device which he (Wilder) typically doesn't like but in this case says are "handled as I've never known them to be by him -- perfectly"; and a beautiful extension of the chorus in the last section that "achieves in its ending a calm, pastoral resolution in the face of the lyric's refusal to be separated from all those loving qualities. A beauty!":
(Wilder, pp. 156-7)

The way you hold your knife, The way we danced till three, The way you changed my life-- No, no! They can't take that away from me! No! They can't take that away from me! ( Kimball, Complete Lyrics, p. 266, hard cover Ed.)

Deena Rosenberg
Fascinating Rhythm: The Collaboration of George and Ira Gershwin,
Ann Arbor: University of Michigan Press, 1991, 1997
(soft cover Ed.)

The integration of "They Can't Take That Away from Me" into the plot of the film is more direct than in the case of most of the movie's other songs. The verse opens with "Our romance won't end on a sorrowful note, / Though by tomorrow you're gone," which speaks directly to the marriage they (Fred's and Ginger's characters) just got so they can be quickly divorced and thereby end the rumors circulating about their being married. It also, and more importantly, reveals to the audience and to themselves that they are in fact in love and don't want to (and won't) get a divorce, presumably ever; so their romance not only "won't end on a sorrowful note"; it won't end at all.

Deena Rosenberg shows how the integration initiated by the original verse is due not only to Ira's lyric but to how the lyric and the music work together. The lyric lets us know the specifics of the memories that will linger and not be taken away and

The melody here has a simple, musing quality; the tonic note is repeated five times to begin each line . . . and then moves to another note in the tonic chord. Except for a hint of blue, under "sip," the harmonies are soothing, in the home key. The rhythm draws listeners into its reverie, starting off the beat, edging us along with easy-going syncopation , providing time for contemplation (or improvisation).

Rosenberg goes on to demonstrate how this correspondence continues throughout the song culminating in the final "A" section where "the lyric makes its most pointed and affecting observations":

"The way you've changed my life" comes pouring out, "life" hitting the sixth note of the scale for the first time in this part of the "A" phrase. The "no, no" 's following it also go higher than before, up to the tonic. These two musical touches immensely heighten the words. ( Rosenberg , p. 148)

Most commentators on Ira Gershwin's lyrics in "They Can't Take That Away from Me" are struck by the contrast between the profound passion conveyed by the song and the simple every day qualities of the images that evoke that passion. Philip Furia explains how Gershwin develops the contrast by contrasting Gershwin to his peers:

Where a lyricist like Hammerstein or Howard Dietz would have strained for poetic images "to remember you by," Gershwin opts for the utterly prosaic: "the way you sip your tea . . . the way you hold your knife . . . the way you sing off key." The lover who notes these trivial details, however, infuses them with understated passion, particularly when he abandons detail altogether with a casual flourish: The mem'ry of all that-- No, no! They can't take that away from me! ( Furia , pp. 145-146)

Michael Feinstein, commenting along similar lines, sees the brilliance in Ira's lyric as the "insight to understand that when you're in love, little things become immensely important and endearing." Feinstein notes that the way Ira moves from the little things or even the ostensibly negative ones to the the "transcendental -- The way you've changed my life" reveals the lyricist's genius (Feinstein, p. 102-103).

Rouben Mamoulian, Armenian-American theater and movie director (He directed both the original Broadway play Porgy as well as George Gershwin's opera Porgy and Bess.) wrote the following in a remembrance of Gershwin in 1938, not long after the composer's death:

George was like a child. He had a child's innocence and imagination. He could look at the same thing ever so many times and yet see it anew every time he looked at it and enjoy it. I remember once we were playing tennis at his home in Beverly Hills. This was after George and Ira had written the score for an Astaire-Rogers picture [Shall We Dance] which had in it the wistful song, "No, No, You Can't Take That Away from Me." Now, George took his tennis, as most of the things he did, close to his heart. He was not a very good player -- I was even worse. Still whenever he missed the ball he was heartbroken -- he would clutch the racket to his chest and moan, "No-o, no-o!" Once, when he missed the ball and moaned "No-o, no-o!" I sang out, "You Can't Take That Away From Me!" George laughed with delight. So after that every time George missed and moaned "No-o, no-o," I came forth with a lusty "You Can't Take That Away From Me!" We did it time and again -- and every time George would burst into laughter as if it had never happened before ( Armitage , pp. 54-55). The way you hold your knife, The way we danced till three, The way you've changed my life-- No, no! They can't take that away from me! No! They can't take that away from me! (the conclusion of the refrain of "They Can't Take That Away from Me" -- Kimball, Complete Lyrics . . . , p. 266) .

On recordings of "They Can't Take That Away from Me" singers either use Ira Gershwin's verse, have dropped it altogether, or have replaced it with one written by someone else. Here is Ira Gershwin's original verse: Our romance won't end on a sorrowful note. Though by Tomorrow you're gone; The song is ended, but as the songwriter wrote, "The melody lingers on." They may take you from me, I'll miss your fond caress. But though they take you from me, I'll still possess: On many other recordings, notably Frank Sinatra on Songs for Young Lovers and Sinatra & Sextet: Live in Paris an alternate verse is used -- winter(s) currently unknown. There are many, many crazy things That will keep me loving you, And with your permission, May I list a few. (Click here to read the full lyric but with the alternate verse as sung by Ella. The shorter, simpler version of the verse serves to detach the lyric from the plot of the film and makes it entirely clear that what is to follow is a list song :

Singers often alter lyrics, and the most common alteration is the omitting of the verse. In this case Ella and Frank probably do this because the substituted version can be understood more easily by someone who hasn't seen the film ( Shall We Dance ) for which the song was written. The listener won't be perplexed by details pertaining to the movie. Ira Gershwin's original verse serves to integrate the song into the plot by commenting on how the couple's relationship is about to end. It tells us she is going to be taken from him; nevertheless, he sings, "Our Romance won't end on a sorrowful note, / Though by tomorrow you're gone; / The song is ended, but as the songwriter * wrote, / The melody lingers on." At this point in the film, their future seems bleak to both of them. None of this is sung in the Fitzgerald or Sinatra versions. In the new verse there is no suggestion of parting. The list of her special qualities that follows isn't something he will have just to remember her by. It serves rather as a catalog of the "many, many crazy things" that will keep him loving her, presumably happily ever after. The new verse is also far easier to sing. Aficionados of The Great American Songbook often seek out recordings of songs that include the verse. In the John Pizzarelli/Jane Monheit performance on the Cafe Songbook Main Stage , John does Ira's original verse. In examples from the Cafe Songbook Record/Video Cabinet , Fred, of course, sings the verse to Ginger just as Ira wrote it; Elvis Costello includes it in his duet with Tony Bennett; Stacey Kent also sings it but changes up its location in the song. Many singers, however, choose not to include the verse at all and begin with the famous first line of the refrain, "The way you wear your hat": Billie Holiday, Peggy Lee, Louis and Ella in their duet, Sinatra on Swingin' Brass and Diana Krall (both live and on Love Scenes ) are in this category. *The songwriter Ira alludes to is, of course, Irving Berlin who wrote " The Song Is Ended " about ten years before the Gershwins wrote "They Can't Take That Away from Me."

• John Pizzarelli and Jane Monheit, TheUntouchable57
• The Music of "Shall We Dance," part 3: omarov
• Fred and Ginger from Shall We Dance: Chatham43
• Billie Holiday: purple0accident
• Peggy Lee: Zedupoca
• Ella Fitzgerald and Louis Armstrong: samarcticmonkey
• Frank Sinatra: AnfitriteNereo
• Tony Bennett and Elvis Costello: Tony Bennett's Vevo Channel
• Diana Krall, Weskoki
• Stacey Kent, Sara092

Notes

• Other songs written for Shall We Dance currently included in the Cafe Songbook Catalog of The Great American Songbook :
• 2. Let's Call the Whole Thing Off
• 3. Shall We Dance?
• 4. Slap That Bass
• 5. They All Laughed
• For a complete listing of songs used in the original production of this movie, see IMDB Soundtrack .
• Written in 1936 for the movie Shall We Dance (1937); Introduced in Shall We Dance by Fred Astaire (See video below.)
• The Ten Movies Co-starring Fred Astaire and Ginger Rogers
• Some Introductory Notes in Word and Video.
• (The documentary appears with the movie itself on the DVD shown below. )
• Edward Jablonski Gershwin A Biography, New York: Doubleday, 1987
• Howard Pollack George Gershwin: His Life and Work Berkeley: Univ. of California Press 2006
• Michael Feinstein, Nice Work If You Can Get It: My Life in Rhythm and Rhyme New York: Hyperion, 1995.
• Edward Jablonski and Lawrence D. Stewart, The Gershwin Years - George and Ira , New York: Doubleday, 1958
• Ira Gershwin, Lyrics on Several Occasions New York: Limelight Editions,1997 (originally published by Knoph, 1959)
• The Gershwins in Hollywood and the Composition of "They Can't Take That Away from Me"
• Ira's Depression and Recovery after George's Death
• *Editor's note: Some of the recordings Ira listened to can be found on the album The Essential Fred Astaire.
• Astaire & Rogers Ultimate Collector's Edition (DVD box set includes all Astaire-Rogers films as well as many special features)
• "They Can't Take That Away from Me": The Second Time Around -- in the movies
• An Alec Wilder Favorite
• Philip Furia, The Poets of Tin Pan Alley: A History of America's Great Lyricists , New York: Oxford University Press, 1990.
• Integration of the Lyrics, Music and Story
• (Ed. Note: See Rosenberg , 345-48 for more on how Ira's lyric and George's music correspond in this song.)
• Merle Armitage, Ed., George Gershwin
• "You Can't Take That Away from Me" on the Tennis Court
• The complete, authoritative lyrics for "They Can't Take That Away from Me" can be found in:
• Robert Kimball, Ed. The Complete Lyrics Of Ira Gershwin , New York: Da Capo Press, 1998.

Cite this page: "They Can't Take That Away from Me": History, Commentary, Discography, Performances on Video, Cafe Songbook, https://greatamericansongbook.net/pages/songs/t/they_cant_take_that_away_f.html