The fourth century was a very significant time in church history. Christianity was legalized in the Roman Empire and began to prosper socially. The emperor proclaimed himself to be a disciple and did many things to promote his new religion. He made Sunday the official day of rest, appointed Christians to high office, exempted the clergy from taxes, built basilicas, financed excavation projects in Palestine, and much more. It was now fashionable to be a Christian. That was not for the better, however, as the church was "secularized" and "paganized." A notable example of this was the celebration of Christ's birth on December 25th.
The celebration of Christ's birth on December 25th was not motivated by Scripture, but by paganism. It was an attempt to "metamorphose" pagan holidays into a Christian observance.
"The Christmas festival was probably the Christian transformation or regeneration of a series of kindred heathen festivals - the Saturnalia, Sigillaria, Juvenalia, and Brumalia - which were kept in Rome in the month of December" (History of the Christian Church, III, p. 396).
The pagan festivals were immensely popular. They were a highly anticipated time of banqueting and excess. That fact surely influenced the decision to celebrate Christ's birth on December 25th. Unable to stamp out the festivities, church leaders tried to tame them and shift the focus to the Christian "Son of God."
The New Testament does not tell us when Christ was born or command us to celebrate the day of His birth. Rather, it says to commemorate His death by eating the Lord's Supper (1 Corinthians 11:23-29). In A.D. 245, Origen commented that only sinners celebrated their birthdays in Scripture, and cited Pharaoh and Herod as his examples (Homilies on Leviticus, VIII). It was not until the fourth century that this practice developed. Christmas was first celebrated at Rome in A.D. 354, at Constantinople in A.D. 379, at Antioch in A.D. 388, and at Alexandria in A.D. 430.
It is not wrong for individual Christians to observe "days" (Romans 14:5-6). That is their right and they should not be judged for doing so. However, the church has no business putting special significance on days that are foreign to Scripture. I hope this helps.