A kitten should stay with his mother until an absolute minimum of 8 weeks, preferably 12 weeks. Even though he is relatively independent by 8 weeks and can survive without his mum, there are social benefits to staying with her and his siblings.
4-5 weeks - Kittens begin eating small amounts of food (which should be soft, such as canned), however, they are still very much dependent on their mother's milk.
6-8 weeks - Kittens are now eating 4 small meals a day, but still nursing from mum.
9-12 weeks - Some kittens may still be nursing at this age, but can survive without her milk. Nursing is mostly for comfort.
Kittens learn cat behaviour from their mum and their brothers and sisters. This includes body language, grooming, litter box habits, how to play and boundaries. There is no better way to learn these skills from family members. Have you ever watched littermates playing? They are rough and tumble, but if a kitten pushes a sibling (or mum) too far, they will be quickly put in their place with a swipe, a bite or a hiss.
Not all cats removed from mum grow into adults who have bad habits, but to reduce your chances, and raise a well-rounded kitten, plenty of time with mum, siblings and humans is an absolute must.
When a kitten is little, he gets antibodies via his mother's milk, these help to keep him safe from infection. From 8 weeks of age, he has a series of vaccinations, spaced 4 weeks apart (8 weeks, 12 weeks, 16 weeks). The reason for this is the antibodies the mother has provided can affect the effectiveness of a single vaccine. A young kitten separated from his mother may not have as much protection as an older cat who has been weaned and fully vaccinated.
My own personal story:
I adopted a 6-week old kitten from some markets many years ago. This poor little lad looked so scared and alone in his cage and I felt the need to get him out of there. So I paid $6 for him and took him home. The second day there he vomited roundworms all over my kitchen. Roundworms can and do happen in kittens, but when they are raised properly, they will be routinely wormed so this kind of thing can't happen. The kitten grew into an adult with several behavioural issues. He had absolutely no idea how to interact with other cats, and would beat them all up, he constantly sprayed, he suffered from petting-induced aggression, which I could manage (I knew the signs), but he took out many of my friends, and he would urinate and defecate all around the house. I truly do believe that all of these problems were due to his being taken away from mum way too early. I don't regret adopting him, he was in a bad place, but until the day he died (at 13), these issues never resolved.
Kittens are kittens for many months, letting yours stay with mum and siblings until 10-12 weeks ensures that you will get a well-rounded kitten, but he will still be young enough to enjoy much more time as a kitten. It really is worth waiting the extra 4 or so weeks to avoid a possible lifetime of behavioural problems.