That's a good question! I did some checking, and according to the Publication Manual of the American Psychological Association, you are correct; while one would normally write out "four" because the rule is to write out numbers one through nine, one should say "4 years" because the number refers to a date. Here are the rules in a nutshell.
Use numbers (numerals) in these instances:
1. For all numbers 10 and above: "There were 17 students in class."
2. All numbers below 10 that are grouped in comparison to numbers 10 and above: "Ony 5 of 17 students passed the course."
3. When using numbers immediately before a unit of measure: "a 5-minute wait"
4. Numbers that represent statistical or mathematical functions or formulas: "a ratio of 12:1"
5. Numbers that represent time, dates, ages, sizes, scores, money, and points on a scale: "It happened 5 years ago"; "a roomful of 6-year-olds"; "$40."
6. Numbers that represent a place in a series: "week 7 of an 8-week diet"
7. In a list of four or more numbers: "We had 1, 2, 5, and 8 pieces, respectively"
For more complete information, you may wish to check the APA manual.
I hope this helps!