You might find Aquinas interesting on this topic, too – he mentions Church Fathers, and Aristotle on the subjects as well. See questions 2-4 here:
http://www.newadvent.org/summa/316800.htm
“Just as man needs bodily rest for the body's refreshment, because he cannot always be at work, since his power is finite and equal to a certain fixed amount of labor, so too is it with his soul, whose power is also finite and equal to a fixed amount of work….Now just as weariness of the body is dispelled by resting the body, so weariness of the soul must needs be remedied by resting the soul: and the soul's rest is pleasure, as stated above….Now such like words or deeds wherein nothing further is sought than the soul's delight, are called playful or humorous. Hence it is necessary at times to make use of them, in order to give rest, as it were, to the soul.….Nevertheless it would seem that in this matter there are three points which require especial caution. The first and chief is that the pleasure in question should not be sought in indecent or injurious deeds or words.”
“In human affairs whatever is against reason is a sin. Now it is against reason for a man to be burdensome to others, by offering no pleasure to others, and by hindering their enjoyment….Now a man who is without mirth, not only is lacking in playful speech, but is also burdensome to others, since he is deaf to the moderate mirth of others. Consequently they are vicious, and are said to be boorish or rude, as the Philosopher [Aristotle] states (Ethic. iv, 8).”