Your app should know nothing about the pipeline
Have your application write to standard output and know nothing else about the pipeline.
Never make your request threads write to a remote logging service directly. The first time that service slows down, your application slows with it, and now logging has caused the outage it existed to explain.
Let an agent on each machine tail that output, batch the lines, and ship them onward. The next stop in any serious pipeline is a buffer.
Why the buffer is there
Understand why the buffer is there. Log volume is spiky and indexing is the slow expensive stage.
When traffic triples during a sale, or a bug starts spewing stack traces at fifty times the usual rate, that buffer absorbs the flood and your indexers drain it at their own pace. Without it the indexer falls behind, agents fill their small local buffers, and lines get dropped precisely during the incident you most need them for.
Choose your storage by which bet you want to make. One popular stack indexes every field of every line, which makes queries fast and storage expensive.
The other takes the opposite bet, indexing only a few labels, storing the raw lines compressed in object storage, and searching them at query time. Queries run slower, and storage costs drop by an order of magnitude, so cost-conscious teams keep choosing it.
Tier your retention, because disks are not free. A common shape is one to two weeks hot and searchable, a month or three in cheaper storage, then a year or more archived for compliance.
Decide those tiers deliberately. The default of keeping everything hot forever is how log storage quietly becomes your biggest line item.
Worked example
Ahead of Black Friday, Deepak's team at an e-commerce company load tests the log pipeline, not just the app. Normal volume is 80,000 lines per second; the test pushes 500,000. Elasticsearch indexing tops out near 200,000, and in the old agent-to-Elasticsearch setup that meant agents dropped lines within 4 minutes. With Kafka in the middle sized for two hours of peak volume, the same test shows indexing lag climbing to 18 minutes and then draining, with zero lines lost. On the actual day, a pricing bug at 09:10 produces a 6x error-log spike on top of peak traffic. Search results run 12 minutes behind for a while, which is annoying, but every line arrives eventually and the postmortem has complete data.