I've removed points that had fewer than 10 attempts. (1 attempt on 4th and 27 last year going for -10 yards probably really tell us anything about football in general.) First and 1 is really tough! But think about how you get first and 1 - you're probably on the 1 yard line - and those goal line plunges tend to lose yardage on average. If one team loses 3 yards on first and 1 and two teams score touchdowns, that still comes out to -0.33 yards per play. Similar logic applies for the other 1st and short distances. 2nd and 3rd downs are pretty consistent, so if teams are throwing deep on 2nd and 2, they aren't connecting often enough to make an impact on the average.
And to wrap up (I hope) this general topic, here are some box plots. In these, the box spans the 25th to 75th percentile of outcomes for passes and rushes for each down and distance. The line across the box is the median, the crosshairs are the mean. I don't know that these provide any particular insight, but I took the time to make the plots, so I might as well post them, dang it! (You can click on them to enlarge them.)
