Because if you have to leave your job to work on your ‘passion’ project then it is not actually your passion it is just an excuse to not do then work that you don’t like.
In fact, if you want to find something that you are passionate about then find something that you cannot ‘not’ do. Which you’ll find the time anyway despite what kind of work you do at your job.
Finding your passion is not the only benefit that you get out of this. You will also learn to do what you want to do in a limited time. This will come handy when your passion project becomes your main thing years later.
Another problem with leaving your job to pursue your passion project is that you will find it easy to push your work to another day to a point where it gets pushed indefinitely.
Or you’ll see yourself accomplishing a lot less by spending a lot more because Parkinson’s Law comes into play. This law states,
Work expands so as to fill the time available for its completion.
If something must be done in six months, it’ll be done in six months. If it must be done in two months, then it will.
So stop dreaming about leaving that cushy or shitty job and making your dreams come true. And get more efficient at doing what you do during the day and build your side hustle into as big as the current job is while working morning or evenings and weekends before you say goodbye to your day job.