The problem with scaling down is that you always must have in mind that not all desktop technologies are adequate for mobile devices and sometimes the “cut” version feels incomplete, unprofessional and that tweaking your desktop website will lead to downloading the big content although it’s not necessary.
The mobile-first approach is a method that will help you to extract the core of your idea and keep in mind what is the most important content, what is the purpose that you want to show. You can use it to increase page’s performance by minimizing unnecessary substance when needed, easier to monitor it. It is also easier to design UI patterns when you can add something tricky, than to reduce something important. Also, it is more helpful to look from accessibility point - how will the slower devices react? The desktop devices are often more powerful than mobile, so it makes easier to code new feature as an addition, not the base which can’t be really used on slower ones. However, it doesn’t mean that mobile design has to be raw and stark. With consideration of the limits, it should be complex and appealing, because both versions are profound for the viewer while browsing, but it doesn’t also mean that the experience has to be identical (in fact it can’t be).
The challenge for mobile views is: how to put as many functionality as possible, with little weight and intuitive experience? Desktop forgives a lot of mistakes - mobile not. If you shoot the bigger quarry first, you will find further developing a lot easier. Remembering about taps and swipes, there is not much to think while converting it to clicking - at least not as much as the other way.
What’s difficult? The most difficult in starting from mobile view, in my opinion, is lack of clear visual design concept. This issue comes strictly from creative, artsy perspective, in which the mobile version sometimes IS a deficient design. Yes, that’s the truth - there is not much space for expression and letting unhampered imagination speak, especially if the design isn’t pure and minimalistic. There is a lot to care about composition of color and shape which has to fit in small space and still look inviting and clear, not to mention the intuitive handling. What’s the solution? I would recommend sketching all of the versions with as many visual ideas as possible and then start the mobile option from a technical and pure user experience and front-end developing view. It seems the best answer for the quandary, a compromise to get only pros from both perspectives!