Build a Small Language Model (SLM) From Scratch

At this current phase of AI evolution, any model with fewer than 1 billion parameters can be called a small language model. If we look at our past models lke GPT-3 which has ~175 billion parameters, and when we think about GPT-4, it is heard that it might contain 1 trilion parameters. As per the scaling law , it says that as parameters increase the performance of model increases. So ofcourse we can infer from here that having large universe architecture can give better results, but what if a small architecture model can give better results than the larger model architecture.