Yeah, you're going to need to climb all the way up to the top of the colossus and stab it. Stab it hard. Its primary weak spot is directly on top of the beast's head, so you're going to have a bit of climbing to do. To begin with, try to circle around the colossus until you reach its rear side; you should be able to see tufts of hair just above its left hoof. When the hoof is level with the ground, dash up to it and jump + grab to hang on.
From here, climbing is just like climbing a cliff, albeit a cliff that moves. As the colossus strides about, you won't be able to move your hero, and in fact might be reoriented a bit due to all the swinging about. Just hang on with your R1 button until things settle down a bit; when the foot is firmly planted on the ground, you'll be able to move as you like.
Now, there is a small dark spot on the leg of the colossus; this is its first weak spot. Technically you can probably stab anywhere in the fur to wound the beast, but you might as well go for the good shot here. When you reach the spot, tap your square button, then tap it again after a second to stab the beast and hamstring it a bit. This won't do much damage to it, but that's not the goal; what really happens is that the leg will drop for a second and the monster will almost let its knee hit the ground, making the calf a horizontal platform for a moment or two. During this time, you have to let go of your hold, stand up, then quickly jump up above the armor on the thigh of the beast and grab a hold of the fur near its...well, near its butt.