The word yoke in Greek is the word zugos ζυγός (dzoo-gos'), which comes from the root of ζεύγνυμι zeugnumi (to join, especially by a "yoke"); a coupling, that is, (figuratively) servitude (a law or obligation); also (literally) the beam of the balance (as connecting the scales): - pair of balances, yoke.
In a literal sense, a yoke was used upon oxen to guide them together as the fields were plowed for farming. Egyptian monuments show a yoke that consisted of a straight bar fastened to the foreheads of the cattle at the root of the horns, and such yokes were no doubt used in Palestine also; but the more usual form was one that rested on the neck (ISBE).
In a figurative sense, the use of "yoke" in the sense of "servitude" is intensely obvious (compare especially Jeremiah 27, 28). Attention needs to be called only to Lamentations 3:27, where "disciplining sorrow" is meant, and to Jeremiah 5:5, where the phrase is a figure for "the law of God." This last use became popular with the Jews at a later period and it is found, e.g. in Apocrypha Baruch 41:3; Psalter of Solomon 7:9; 17:32; Ab. iii. 7,. and in this sense the phrase is employed. by Christ in Matthew 11:29 f. "My yoke" here means "the service of God as I teach it" (the common interpretation, "the sorrows that I bear," is utterly irrelevant) and the emphasis is on "my." The contrast is not between "yoke" and "no yoke," but between "my teaching" (light yoke) and "the current scribal teaching'; (heavy yoke) (ISBE).
Undestanding the difference between the two forms of yoke, helps us to understand this more clearly.
John MacArthur writes in his sermon titled, "Jesus' Personal Invitation, Part 2", "The Jews used the phrase “the yoke” to refer to entering into submission to something. That’s why Peter used it in Acts as I read earlier about the yoke that the Pharisees put on people that they can’t bear.
Acts 15:10 - Now therefore, why do you test God by putting a yoke on the neck of the disciples which neither our fathers nor we were able to bear?
"The yoke is that which causes submission. And Jesus says When you come to Me you must come with a submissive heart. You must come to take a yoke."
MacArthur further adds, "in extrabiblical writings, for example, Sirach records this statement, “Put your neck under the yoke and let your soul receive instruction.” And that pretty well typifies how the Jews saw the yoke. It was a yoke of instruction. They talked about the yoke of the torah...the yoke of the law, the yoke of the commandments, the yoke of God. And that is what our Lord is asking them to do, He is saying, take My yoke upon you. And then He adds this phrase: “And learn from Me,” it is a yoke of submission to His Lordship, to His teaching, to His instruction. It is a yoke that implies obedience. And I believe true salvation occurs when you in your desperation turn from sin to Christ with a willingness to have Him take control of your life. I don’t think, therefore, that you can take Jesus as Savior and not as Lord."
It is now a bit easier to undestand what Jesus is saying in this passage and what he was speaking of specifically.
The Law, the Torah, was a burden for people to follow, especially when there were 613 laws that were established and it was nearly impossible for anyone to be found perfect concerning obedience to the Law.
Jesus said in Matthew 5:17, "Think not that I am come to destroy the law, or the prophets: I am not come to destroy, but to fulfil."
Where we could not fulfill the requirements of the Law, Jesus did. He did not destroy those requirements, He fulfilled them. All of them.
Come unto me, all ye that labour and are heavy laden, and I will give you rest. Take my yoke upon you, and learn of me; for I am meek and lowly in heart: and ye shall find rest unto your souls. For my yoke is easy, and my burden is light. Matthew 11:28-30, King James Version.
Remember the thief on the cross who was crucified next to Jesus? The one with a repentant heart, recognized that he deserved the just punishment of his sin by being crucified and he asked Jesus to remember him when He entered into his kingdom. Jesus then followed with His reply saying, "Assuredly, I say to you, today you will be with me in paradise." (Luke 23:43, NKJV)
As I read other commentaries regarding this passage, I found this one from Matthew Henry's Concise Commentary the most profound. When we closely study the Scripture, especially what Jesus is saying in Matthew 11:28-30, He is talking about giving rest to the soul that has been burdened by the performance of attempting to keep the Law, the Ten Commandments and the burden to feel that an adequate amount of good works will suffice for God's approval and their entrance into His eternal Kingdom.
"Those who labour to establish their own righteousness also labour in vain. The convinced sinner is heavy-laden with guilt and terror; and the tempted and afflicted believer has labours and burdens. Christ invites all to come to him for rest to their souls. He alone gives this invitation; men come to him, when, feeling their guilt and misery, and believing his love and power to help, they seek him in fervent prayer. Thus it is the duty and interest of weary and heavy-laden sinners, to come to Jesus Christ. This is the gospel call; Whoever will, let him come. All who thus come will receive rest as Christ's gift, and obtain peace and comfort in their hearts. But in coming to him they must take his yoke, and submit to his authority. They must learn of him all things, as to their comfort and obedience. He accepts the willing servant, however imperfect the services. Here we may find rest for our souls, and here only. Nor need we fear his yoke. His commandments are holy, just, and good. It requires self-denial, and exposes to difficulties, but this is abundantly repaid, even in this world, by inward peace and joy. It is a yoke that is lined with love. So powerful are the assistances he gives us, so suitable the encouragements, and so strong the consolations to be found in the way of duty, that we may truly say, it is a yoke of pleasantness. The way of duty is the way of rest. The truths Christ teaches are such as we may venture our souls upon. Such is the Redeemer's mercy; and why should the labouring and burdened sinner seek for rest from any other quarter? Let us come to him daily, for deliverance from wrath and guilt, from sin and Satan, from all our cares, fears, and sorrows. But forced obedience, far from being easy and light, is a heavy burden. In vain do we draw near to Jesus with our lips, while the heart is far from him. Then come to Jesus to find rest for your souls."
Matthew Henry's Concise Commentary, Matthew 11:25-30.
International Standard Bible Encyclopedia, e-Sword.
MacArthur, John. Jesus' Personal Invitation, Part 2. Retrieved on July 10, 2009 from http://www.gty.org/Resources/Sermons/2289
Meyers, R. (2006). e-Sword. The sword of the Lord with an electronic edge (Verson 7.8) [Computer software].