Windows 服务增加 CPU 消耗
在我的工作中，我负责六个 Windows 服务，这些服务是用 C# 2003 编写的。每个服务都包含一个计时器，大约每分钟触发一次，这是它们的大部分工作发生的时间。
我的问题是，随着这些服务的运行，它们开始在循环的每次迭代中消耗越来越多的 CPU 时间，即使它们没有任何有意义的工作要做（即，它们只是闲置，查看数据库去做某事）。 当它们启动时，每个服务平均使用 4 个 CPU 的（大约）2-3%，这很好。 24 小时后，每个服务将在其循环运行期间消耗整个处理器。
有人可以帮忙吗？ 我不知道是什么原因造成的。 我们当前的解决方案是每天重新启动服务一次（它们会自行关闭，然后脚本发现它们已离线并在凌晨 3 点左右重新启动它们）。 但这不是一个长期的解决方案； 我担心的是，随着服务变得更加繁忙，每天重新启动一次它们可能还不够......但是，由于存在显着的启动损失（它们都使用 NHibernate 进行数据访问），当它们变得更加繁忙时，这正是我们不想做的就是更频繁地重新启动它们。
@akmad：确实，这非常困难。
- 是的，随着时间的推移，单独运行的服务会表现出相同的症状。
- 不，事实并非如此。 我们已经研究过了。 这可能发生在上午 10 点、下午 6 点或半夜。 没有一致性。
- 我们的确是; 他们确实如此。 这些服务正在做它们应该做的事情，没有别的。
- 不幸的是，这需要准确地预先知道服务何时会耗尽 CPU，这种情况发生在一个不可预测的时间表上，而且从来不会很快......这使事情变得更加困难，因为我的老板会在他们开始使用它们时运行并重新启动它们。问题而不考虑调试问题。
- 不，他们使用相当一致的 RAM 量（每个大约 60-80MB，机器上有 4GB）。
很好的建议，但请放心，我们已经尝试了所有常见的故障排除方法。 我希望这是一个有人可能知道的 .NET 问题，我们可以努力解决它。 我老板的解决方案（我坚决不想实现）是在数据库中放置一个字段，该字段保存服务在白天重新启动的多次，这样他就可以让问题消失而不用去想它。 我正在拼命寻找真正问题的原因，以便解决它，因为该解决方案将在大约六个月内变成一场灾难。
@Yaakov Ellis：它们各自有不同的功能。 从异地某处的 Oracle 数据库中读取记录； 另一个处理这些记录并将属于这些记录的文件传输到我们的系统； 第三个检查这些文件以确保它们符合我们的预期； 另一个是维护服务，它不断检查磁盘空间（我们有足够的空间）等内容并轮询其他服务器以确保它们处于活动状态； 其中一个的运行只是为了确保所有其他的都在运行并完成其工作，监视和报告错误，并重新启动任何无法保持整个系统每天 24 小时运行的东西。
所以，如果你问的是我认为你在问的问题，不，所有这些服务所做的事情（除了通过 NHibernate 进行数据库访问之外）没有一件我可以指出的潜在问题。 不幸的是，如果这确实是真正的问题（这不会让我大吃一惊），那么整个事情可能就会搞砸——我最终将用简单的 SQL 重写所有这些。 我希望这是一个垃圾收集器问题或者比 NHibernate 更容易处理的问题。
@Joshdan：没有秘密。 正如我所说，我们已经尝试了所有常见的故障排除方法。 分析没有任何帮助：我们使用的分析器无法指向 CPU 使用率较高时实际执行的任何代码。 大约一个月前，为了寻找这个问题，这些服务被拆散了。 我们对代码的每一部分进行了分析，试图找出问题是否出在我们的代码上； 我不是来这里问的，因为我还没做作业。 如果这是一个服务执行比预期更多的工作的简单案例，那么这种情况就会被捕获。
这里的问题是，大多数时候，服务根本不做任何事情，但仍然设法消耗四个 CPU 核心的 25% 或更多：他们发现没有工作要做，并退出循环并等待下一次迭代。 从字面上看，这几乎不占用任何 CPU 时间。
下面是我们看到的一个行为示例，该服务在两天内没有任何工作可做（在不变的环境中）。 这是上周拍摄的：
第一天上午 8 点：平均。 CPU使用率约3%
第 1 天，下午 6 点：平均。 CPU使用率约8%
第 2 天上午 7 点：平均。 CPU使用率约20%
第 2 天上午 11 点：平均。 CPU 使用率约为 30%
在查看了所有可能的常见原因后，我在这里提出了这个问题，因为我认为（事实证明是正确的）我会得到更具创新性的答案（如 Ubiguchi 的答案）或指针一些我没有想到的事情（比如 Ian 的建议）。
CPU 峰值是否会发生 紧接在计时器之前 回调，在计时器回调内， 或立即跟随计时器 回调？
你误会了。 这不是尖峰。 如果是的话，那就没有问题了； 我可以应对尖峰。 但事实并非如此……CPU 使用率普遍上升。 即使服务什么也不做，也在等待下一个计时器命中。 当服务启动时，一切都很好，很平静，图表看起来就像您所期望的那样......通常，使用率为 0%，当 NHibernate 访问数据库或服务执行一些琐碎的工作时，使用率会飙升至 10% 。 但在进程运行时，这会始终增加到 25% 的整体使用率（如果我让它走得太远的话，会增加更多）。
这使得 Ian 的建议成为合乎逻辑的灵丹妙药（NHibernate 在你不注意的时候做了很多事情）。 唉，我已经实现了他的解决方案，但它没有产生效果（我没有证据证明这一点，但我实际上认为这让事情变得更糟......平均使用率似乎上升现在快多了）。 请注意，删除 NHibernate“部分”（如您所建议的）是不可行的，因为这会删除服务中大约 90% 的代码，这将使我排除计时器的问题（我绝对打算这样做）尝试），但不能帮助我排除 NHibernate 的问题，因为如果 NHibernate 导致了这个问题，那么所实施的狡猾的修复（见下文）将不得不成为系统的工作方式； 我们在这个项目中如此依赖 NHibernate，以至于 PM 根本不会接受它导致了无法解决的结构性问题。
我刚刚注意到一种绝望感 问题——你的问题 除非出现小奇迹，否则将继续下去
。但这并不意味着它会以这种方式结束。 目前，这些服务每天都会重新启动（可以选择输入一天中任意几个小时来关闭和重新启动），这可以修复问题，但一旦进入生产机器，就不能成为长期解决方案并开始变得忙碌。 无论我解决它们还是 PM 维持对它们的限制，这些问题都不会继续下去。 显然，我更愿意实施真正的修复，但由于初始测试没有发现任何原因，并且服务已经经过广泛审查，PM 宁愿让它们多次重新启动，也不愿花更多时间尝试修复它们。 这完全超出了我的控制范围，这让你所说的奇迹变得更加重要。
这非常有趣（到目前为止 因为您信任您的分析器）。
我不。 但是，这些是用 .NET 1.1 编写的 Windows 服务，运行在 Windows 2000 计算机上，由狡猾的 Nant 脚本部署，使用旧版本的 NHibernate 进行数据库访问。 那台机器上几乎没有什么我可以说我真正信任的。
At my job, I have a clutch of six Windows services that I am responsible for, written in C# 2003. Each of these services contain a timer that fires every minute or so, where the majority of their work happens.
My problem is that, as these services run, they start to consume more and more CPU time through each iteration of the loop, even if there is no meaningful work for them to do (ie, they're just idling, looking through the database for something to do). When they start up, each service uses an average of (about) 2-3% of 4 CPUs, which is fine. After 24 hours, each service will be consuming an entire processor for the duration of its loop's run.
Can anyone help? I'm at a loss as to what could be causing this. Our current solution is to restart the services once a day (they shut themselves down, then a script sees that they're offline and restarts them at about 3AM). But this is not a long term solution; my concern is that as the services get busier, restarting them once a day may not be sufficient... but as there's a significant startup penalty (they all use NHibernate for data access), as they get busier, exactly what we don't want to be doing is restarting them more frequently.
@akmad: True, it is very difficult.
- Yes, a service run in isolation will show the same symptom over time.
- No, it doesn't. We've looked at that. This can happen at 10AM or 6PM or in the middle of the night. There's no consistency.
- We do; and they are. The services are doing exactly what they should be, and nothing else.
- Unfortunately, that requires foreknowledge of exactly when the services are going to be maxing out CPUs, which happens on an unpredictable schedule, and never very quickly... which makes things doubly difficult, because my boss will run and restart them when they start having problems without thinking of debug issues.
- No, they're using a fairly consistent amount of RAM (approx. 60-80MB each, out of 4GB on the machine).
Good suggestions, but rest assured, we have tried all of the usual troubleshooting. What I'm hoping is that this is a .NET issue that someone might know about, that we can work on solving. My boss' solution (which I emphatically don't want to implement) is to put a field in the database which holds multiple times for the services to restart during the day, so that he can make the problem go away and not think about it. I'm desperately seeking the cause of the real problem so that I can fix it, because that solution will become a disaster in about six months.
@Yaakov Ellis: They each have a different function. One reads records out of an Oracle database somewhere offsite; another one processes those records and transfers files belonging to those records over to our system; a third checks those files to make sure they're what we expect them to be; another is a maintenance service that constantly checks things like disk space (that we have enough) and polls other servers to make sure they're alive; one is running only to make sure all of these other ones are running and doing their jobs, monitors and reports errors, and restarts anything that's failed to keep the whole system going 24 hours a day.
So, if you're asking what I think you're asking, no, there isn't one common thing that all these services do (other than database access via NHibernate) that I can point to as a potential problem. Unfortunately, if that turns out to be the actual issue (which wouldn't surprise me greatly), the whole thing might be screwed -- and I'll end up rewriting all of them in simple SQL. I'm hoping it's a garbage collector problem or something easier to deal with than NHibernate.
@Joshdan: No secret. As I said, we've tried all the usual troubleshooting. Profiling was unhelpful: the profiler we use was unable to point to any code that was actually executing when the CPU usage was high. These services were torn apart about a month ago looking for this problem. Every section of code was analyzed to attempt to figure out if our code was the issue; I'm not here asking because I haven't done my homework. Were this a simple case of the services doing more work than anticipated, that's something that would have been caught.
The problem here is that, most of the time, the services are not doing anything at all, yet still manage to consume 25% or more of four CPU cores: they're finding no work to do, and exiting their loop and waiting for the next iteration. This should, quite literally, take almost no CPU time at all.
Here's a example of behaviour we're seeing, on a service with no work to do for two days (in an unchanging environment). This was captured last week:
Day 1, 8AM: Avg. CPU usage approx 3%
Day 1, 6PM: Avg. CPU usage approx 8%
Day 2, 7AM: Avg. CPU usage approx 20%
Day 2, 11AM: Avg. CPU usage approx 30%
Having looked at all of the possible mundane reasons for this, I've asked this question here because I figured (rightly, as it turns out) that I'd get more innovative answers (like Ubiguchi's), or pointers to things I hadn't thought of (like Ian's suggestion).
So does the CPU spike happen
immediately preceding the timer
callback, within the timer callback,
or immediately following the timer
callback?
You misunderstand. This is not a spike. If it were, there would be no problem; I can deal with spikes. But it's not... the CPU usage is going up generally. Even when the service is doing nothing, waiting for the next timer hit. When the service starts up, things are nice and calm, and the graph looks like what you'd expect... generally, 0% usage, with spikes to 10% as NHibernate hits the database or the service does some trivial amount of work. But this increases to an across-the-board 25% (more if I let it go too far) usage at all times while the process is running.
That made Ian's suggestion the logical silver bullet (NHibernate does a lot of stuff when you're not looking). Alas, I've implemented his solution, but it hasn't had an effect (I have no proof of this, but I actually think it's made things worse... average usage is seeming to go up much faster now). Note that stripping out the NHibernate "sections" (as you recommend) is not feasible, since that would strip out about 90% of the code in the service, which would let me rule out the timer as a problem (which I absolutely intend to try), but can't help me rule out NHibernate as the issue, because if NHibernate is causing this, then the dodgy fix that's implemented (see below) is just going to have to become The Way The System Works; we are so dependent on NHibernate for this project that the PM simply won't accept that it's causing an unresolvable structural problem.
I just noted a sense of desperation in
the question -- that your problems
would continue barring a small miracle
Don't mean for it to come off that way. At the moment, the services are being restarted daily (with an option to input any number of hours of the day for them to shutdown and restart), which patches the problem but cannot be a long-term solution once they go onto the production machine and start to become busy. The problems will not continue, whether I fix them or the PM maintains this constraint on them. Obviously, I would prefer to implement a real fix, but since the initial testing revealed no reason for this, and the services have already been extensively reviewed, the PM would rather just have them restart multiple times than spend any more time trying to fix them. That's entirely out of my control and makes the miracle you were talking about more important than it would otherwise be.
That is extremely intriguing (insofar
as you trust your profiler).
I don't. But then, these are Windows services written in .NET 1.1 running on a Windows 2000 machine, deployed by a dodgy Nant script, using an old version of NHibernate for database access. There's little on that machine I would actually say I trust.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（7）
我的感觉是，无论根本原因多么奇怪，通常的故障排除步骤都是找到问题的最佳选择。
由于这是一个性能问题，因此良好的测量非常宝贵。 整个进程 CPU 使用率的测量范围过于宽泛。 您的服务将时间花在哪里？ 您可以使用探查器来测量这一点，或者只记录各个部分的开始和停止。 如果您连这一点都做不到，那么请使用 Andrea Bertani 的建议——通过删除其他部分来隔离部分。
一旦确定了大致区域，就可以进行更细粒度的测量，直到找出 CPU 使用率的来源。 如果此时如何解决它并不明显，那么您至少有解决更具体问题的弹药。
如果您实际上已经完成了所有这些常见的故障排除，请告诉我们这个秘密。
My feeling is that no matter how bizarre the underlying cause, the usual troubleshooting steps are your best bet for locating the issue.
Since this is a performance issue, good measurements are invaluable. The overall process CPU usage is far too broad a measurement. Where is your service spending its time? You could use a profiler to measure this, or just log various section start and stops. If you aren't able to do even that, then use Andrea Bertani's suggestion -- isolate sections by removing others.
Once you've located the general area, then you can make even finer-grained measurements, until you sort out the source of the CPU usage. If it's not obvious how to fix it at that point, you at least have ammunition for a much more specific question.
If you have in fact already done all this usual troubleshooting, please do let us in on the secret.
我从这里开始：
关于可能性的一些想法：
Here's where I'd start:
A couple of thoughts on possibilities:
您提到您正在使用 NHibernate - 您是否在适当的时间点关闭 NHibernate 会话（例如每次迭代结束时？）
如果没有，那么加载到内存中的对象映射的大小将随着时间的推移逐渐增加，并且每个会话刷新将花费越来越多的 CPU 时间。
You mentioned that you're using NHibernate - are you closing your NHibernate sessions at appropriate points (such as the end of each iteration?)
If not, then the size of the object map loaded into memory will be gradually increasing over time, and each session flush will take increasingly more CPU time.
远程调试未知的应用程序显然相当困难...但我会考虑以下一些内容：
祝你好运！
It's obviously pretty difficult to remotely debug you're unknown application... but here are some things I'd look at:
Best of luck!
恐怕这个答案只是建议您查看一些方向，但是在 .NET Windows 服务中看到类似的问题后，我有一些想法可能会对您有所帮助。
我的第一个建议是，您的服务可能在处理内存的方式或处理非托管内存的方式上存在一些错误。 上次我追踪到类似的问题时发现，我们正在使用第 3 方 OSS 库来存储静态内存中非托管对象的句柄。 服务运行的时间越长，服务获取的句柄就越多，这导致进程的 CPU 性能迅速下降。 尝试解决此类问题的方法是确保您的服务在计时器调用之间不会在内存中存储任何内容，但如果您的第 3 方库使用静态内存，您可能需要采取一些巧妙的措施，例如为计时器调用创建一个应用程序域并放弃处理完成后，应用程序 doamin（及其静态内存）。
我在类似情况下看到的另一个问题是计时器同步代码受到怀疑，这实际上允许多个线程同时运行处理代码。 当我们调试代码时，我们发现第一个线程正在阻塞第二个线程，当第二个线程启动时，第三个线程被阻塞。 随着时间的推移，阻塞持续的时间越来越长，CPU 使用率因此达到最高点。 我们用来解决该问题的解决方案是实现适当的同步代码，以便计时器仅在不被阻止的情况下启动另一个线程。
希望这会有所帮助，但如果我的想法都是转移注意力的话，请预先道歉。
'Fraid this answer is only going to suggest some directions for you to look in, but having seen similar problems in .NET Windows Services I have a couple of thoughts you might find helpful.
My first suggestion is your services might have some bugs in either the way they handle memory, or perhaps in the way they handle unmanaged memory. The last time I tracked down a similar issue it turned out a 3rd party OSS libray we were using stored handles to unmanaged objects in static memory. The longer the service ran the more handles the service picked up which caused the process' CPU performance to nose-dive very quickly. The way to try and resolve this sort of issue to ensure your services store nothing in memory inbetween the timer invocations, although if your 3rd party libraries use static memory you might have to do something clever like create an app domain for the timer invocation and ditch the app doamin (and its static memory) once processing is complete.
The other issue I've seen in similar circumstances was with the timer synchronization code being suspect, which in effect allowed more than one thread to be running the processing code at once. When we debugged the code we found the 1st thread was blocking the 2nd, and by the time the 2nd kicked off there was a 3rd being blocked. Over time the blocking was lasting longer and longer and the CPU usage was therefore heading to the top. The solution we used to fix the issue was to implement proper synchronization code so the timer only kicked off another thread if it wouldn't be blocked.
Hope this helps, but apologies up front if both my thoughts are red herrings.
我建议将问题分解。
首先，找到一种 100% 快速重现问题的方法。 降低计时器，以便服务更频繁地启动（例如，比正常情况快 10 倍）。 如果问题出现的速度快了 10 倍，那么它与迭代次数有关，而不是与实时或服务完成的实际工作有关）。 您将能够比每天一次更快地执行后续步骤。
其次，注释掉所有真正工作的代码，只注释服务、定时器和同步机制。 如果问题仍然出现，则问题将出现在代码的该部分中。
如果没有，则开始添加回您注释掉的代码，一次添加一段。 最终，您应该找出代码的哪一部分导致了问题。
I suggest to hack the problem into pieces.
First, find a way to reproduce the problem 100% of the times and quickly. Lower the timer so that the services fire up more frequently (for example, 10 times quicker than normal). If the problem arises 10 times quicker, then it's related to the number of iterations and not to real time or to real work done by the services). And you will be able to do the next steps quicker than once a day.
Second, comment out all the real work code, and let only the services, the timers and the synchronization mechanism. If the problem still shows up, than it will be in that part of the code.
If it doesn't, then start adding back the code you commented out, one piece at a time. Eventually, you should find out what part of the code is causing the problem.
听起来像是计时器的线程问题。 您可能有一个工作单元阻塞了在不同工作线程上运行的另一个工作单元，导致每次计时器触发时它们都会堆积起来。 或者，您可能会遇到比您预期的生活和工作时间更长的实例。
我建议重构计时器。 将其替换为在线程池上排队工作的单个线程。 您可以使用 Sleep() 线程来控制它查找新工作的频率。 确保这是您的代码是多线程的唯一地方。 所有其他对象都应在工作准备好处理时实例化，并在工作完成后销毁。 状态是多线程代码中的敌人。
设计缺乏的另一个领域似乎是您有多个服务正在轮询资源来执行某些操作。 我建议将它们统一在一个服务下。 他们可能会做不同的事情，但他们会齐心协力； 您只是使用文件系统、数据库等来替代方法调用。 还有，2003年？ 我为你感到伤心。
Sounds like a threading issue with the timer. You might have one unit of work blocking another running on different worker threads, causing them to stack up every time the timer fires. Or you might have instances living and working longer than you expect.
I'd suggest refactoring out the timer. Replace it with a single thread that queues up work on the ThreadPool. You can Sleep() the thread to control how often it looks for new work. Make sure this is the only place where your code is multithreaded. All other objects should be instantiated as work is readied for processing and destroyed after that work is completed. STATE IS THE ENEMY in multithreaded code.
Another area where the design is lacking appears to be that you have multiple services that are polling resources to do something. I'd suggest unifying them under a single service. They might do seperate things, but they're working in unison; you're just using the filesystem, database, etc as a substitution for method calls. Also, 2003? I feel bad for you.