Over the last 2 weeks I’ve had problems on three sites with the exact same problem.
The watchdog table grew FASTER than the cron job would have deleted it.
(the cron job should delete old log entries older than e.g. 3 days)
PLUS the PUBLIC site display was full of error messages, which could not be logged to the database anymore
Warning: The table ‘mf_watchdog’ is full query: INSERT INTO mf_watchdog (uid, type, message, severity, link, location, referer, hostname, timestamp) VALUES (2, ‘php’, ‘Table 'mf_dpprod5.mf_cache_views' doesn't exist\nquery: LOCK TABLES mf_cache_views WRITE in /home/drupal/includes/database.mysql.inc on line 172.’, 2, ‘’, ‘http://www.marketingfan.com/admin/logs/watchdog’, ‘http://www.marketingfan.com/’, ’2.4.10.8’, 1184764208) in /home/drupal/includes/database.mysql.inc on line 172
This again ended in a “device full” last night when one watchdog table became 5 gigs.
(and I had a “table full” problem on another site last week – tough 64bit MYSQL should be able to do 4TB spaces, that one was 4GB only)
Now, it’s obvious that disk full problems shouldn’t be a problem for a content management system like Drupal to take care about,
but the system or application administrators. However, this particular situation IS an application problen inherent to Drupal.
Because, the REASON the table grew so fast was, that the certain table “cache” was marked as crashed/corrupt in mysql,
and hence EVERY httpp request to the Drupal site caused several tries to read/write from/to the cache table – resulting in an error being logged.
One must agree that it makes NO SENSE to dump a “table …. marked as crashed” several hundred times PER SECOND into the watchdog,
and I therefore already urgently requested a redesign of some kind of alert threshold.
So my idea is, if an error message was sent 100 times to the watchdog in the last hour (which could have been seen by a human), chances are the next 120,000 entries in that hour won’t add any information to it, but just fill it up with crap. So I would say we supress this exact same watchdog entry… (yaddaa yadda – it’s loss of useless information – because the table IS still corrupt… no matter if it happens at 1:31 or 1:35)
Another nice feature I think will be required would have various levels definable by a) severity of the error b) component (i.e. PHP vs. applciation errors) where the error arrives.
Whatever solution deems appropriate to the Drupal core maintainers, the current situation of Drupal filling it’s own watchdog with useless entries
leading to a halting server (because mysql tables after often placed on /var partitions, which also carry system log file)
is not an operable solution but a hazard for production systems.
Fortunately the watchdog has been the target of several improvements recently to move Drupal more to an enterprise-class application, so I’ve got bit hopes that this flaw will
find it’s way into Drupal some day as well.
Let me know what you think.
Did your Content Management System ever fill up the whole disk of the server?
Comments
Drupal 6
This problem will be fixed in Drupal 6 -- in Drupal 6 the watchdog table is pruned when there are more than x messages instead of keeping the messages around of y days. That should improve the situation -- but doesn't necessarily fix all edge cases.
However, you can configure Drupal 6 to use syslog, in which case it would log watchdog messages to your system's log manager (i.e. syslogd) rather than to Drupal's database.
what an honor - thanks for the fast confirmation, but...
Hey Dries,
what an honor to have you commenting here!
Thanks for the quick reply (how did you find my post so fast?)
and thanks for the confirmation you are on this.
IMHO your solution would cure the "device full" problem,
but would also leave the system "dumping same error" like crazy.
Last night at 2.15 I also just did a truncate table on the watchdog table to
get my server back, but I assume there were some pre-conditions logged that
I'd love to see today to mabe dig into the intial problem before that cache table went corrupt
The SYSLOG change is a great enhancement in anycase, and I think having that
configurable my "source" (ie MYSQL/PHP) would do just what we require - send problems to the sys managers that they can take care about, but keep an eye on the app functionality in Drupal
Not sure if Barcelona is too late for the release, but I'll be happy to discuss my thoughts and experiences with other large scale logger's then :-)
see you in BCN
thanks
christoph
Watchdog
This kind of discussion about real problems like yours with watchdog will be very welcome at the Drupalcon. Experience is knowledge.
See you,
Paco
theborg@drupal.org
theborg@drupal.cat
drupalcon
Oh yes, experience is knowledge...
so we can catchup in Barcelona?
drupalcon
Sure, the beer here is not as good as yours but you're invited anyway.
Cheers
nice.... still looking for a good place
Nice, got the flight, but I'm still looking for a nice place to stay... inside the city near Rambla would be nice, but imho going out at night needs cabs anyway, so I'd rather pick one near the conference place, or?
cheers
c
Where to stay
I posted a map to get an idea and something about where to stay at the forum:
http://barcelona2007.drupalcon.org/node/15#comment-140
The city has grown in hotel rooms (and prices) in the last few years so the offer is better now.
The venue is a very good place to organize conferences but is outside Barcelona, in a town that belogns to the metropolitan area. I would not recommend to stay there if you want to visit the city.
Barcelona is not a big city and if you stay near Ramblas, you can go out at nights and get to the hotel by walking or public transportation, no cabs needed. Everything from bars, clubs, restaurants or the beaches are near there, also is the old part so you can visit monuments like the cathedral, the city council, the Picasso musem, etc. Although Ciutat Vella has two faces, the modern of Born neighbours and the ghetto that is becoming the rest of the district. I have to say that, looking like a tourist, you have to take good care of your bag, camera, etc. because of the robatories but I think that this is common in other cities
If you want to stay near everything, I would look for a hotel in the Eixample district near the tram that goes to the Citilab, for example the NH Les Corts. Being there you can go direct to the venue by tram and to the Ramblas, Paseig de Gràcia or Rambla Catalunya, (for me the most beautiful street of Barcelona) by bus.
Map to look for hotels
Gracia district is also a nice place to stay just over the Paseig de Gràcia, is an old neighbourhood, with shops, bars and restaurants.
I had the luck of staying with my wife at Vienna's Marriott two years ago, from there we moved inside the city by tram, metro and bus and it was wonderful. Here Sarrià is the neighbourhood of the rich people but I think that the hotels will be expensive too.
If you want to know more about the hotel you are going, send me an email and I will help you. The drupal.cat people will be very happy to help everybody that comes to the DrupalCon.
Good luck,
Paco