MongoDB collections can be capped, even though we haven't added the UI for it yet #660460: Watchdog: Support Capped Collections, just go into the mongo shell:
mongo> db.watchdog.drop();mongo> db.createCollection("watchdog", {capped:true, size:1000000, max:10000});
If you do this, we have the problem that a one error message that happens frequently can "spam" the log and quickly become the only error in the log. Consider a php error that happens on cron or in drush, as you iterate over 10,000 rows, or consider a high traffic site with a php error.
To solve this problem, I propose adding a count, and line number to the log, and that if a new error occurs within a few seconds of another error from the same line number, that we just increment the count, and not write a new line. Per chx's suggestion, I'm sacrificing accuracy a little here... it would be more accurate to test for file and line number, but since this record is written alot, the suggestion is to only use an index on line number, and on the rare case that we have a line number conflict, that we just lose the data. Watchdog isn't critical data, so the assumption is that this is OK.