I have been working in application development on embedded devices for a few years now. All the embedded platforms that we work on do not have a good development environment. Some platforms don’t even have GDB. So we have to fully rely on logs to analyse the issues.
When the only way to debug your code is through logs, some people become lazy and don’t log judiciously. Instead, a lot of information is logged, irrespective of the importance of the information or the frequency of logs. When you work in a team of 15+ developers(in which some are lazy!) the logs can easily bloat up. In a recent project, we had so many logs that the logging rate was ~2 Mega Bytes per minute. It is a nightmare to comb through thousands of lines of logs to find the logs you are interested in and analyse issues.
Following are some ways in which we improved our situation.