In S1E14, Emily and Richard are on the phone with a realtor discussing buying a new property at Martha’s Vineyard, and Emily wants to take the deal even though the grounds fees are higher. Richard holds out, however, and their realtor agrees to negotiate the grounds fees. Richard hangs up looking very pleased with himself, and Emily says, “Careful Richard, that canary you ate is going to spoil your dinner.”
“You look like the cat that ate the canary” is a saying used to describe someone who looks very smug and self-satisfied.
This isn’t really a cultural reference, but I personally wasn’t familiar with that idiom!