您使用什么样的保护措施来避免意外地对生产环境进行意外更改?

发布于 2024-07-23 10:54:53 字数 423 浏览 4 评论 0原文

因为我们没有良好的暂存环境,所以我们经常必须调试生产系统上的问题。 我们有网络、应用程序和数据库服务器。

执行此操作时,您使用什么样的保护措施来避免意外地对生产环境进行意外更改?


编辑:

该应用程序是一个非常复杂的 B2B 垂直 Web 应用程序。 这涉及到很多数据。 有些表有接近 1 亿条记录。


编辑:

我们现有的临时环境没有镜像生产的能力。 除了实际的数据库数据之外,还涉及数百GB的数据文件。


编辑:

我们确实对代码使用源代码控制,但不对存储过程使用源代码控制。 源代码管理中有一些旧的存储过程,但没有人再对其进行更新。

主要关注的是文件系统上的数据库和数据。

顺便说一句,我是这家公司的顾问,而不是实际的员工。

Because we don't have a good staging environment we often have to debug issues on our production systems. We have web, application, and database servers.

What kind of safeguards do you use to avoid accidentally making unintended changes to your production environment when doing this?


EDIT:

The application is a very complex B2B vertical web application. There is a lot of data involved. Some tables have close to 100 million records.


EDIT:

The staging environment we have in place does not have the capacity to mirror production. There are also hundreds of gigabytes of data files involved besides the actual database data.


EDIT:

We do use source control for the code but not for the stored procedures. There are some old stored procedures in source control but nobody keeps that updated anymore.

The main concerns are the database and data on the file system.

BTW, I am a consultant at this company, not an actual employee.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(10

祁梦 2024-07-30 10:54:53

最直接的答案是:“不要这样做。”

The most direct answer is: "Don't do that."

纵情客 2024-07-30 10:54:53

源头控制。 当事情出现无法挽回的错误时,没有什么比回滚更好的了。 此外,差异可以帮助您将更改复制到其他生产系统。

source control. nothing like a rollback when things to irreparably wrong. Also, a diff can help you replicate the changes to other production systems.

段念尘 2024-07-30 10:54:53

新的生产版本通过我们的系统人员进行,程序员和开发人员只能请求让他们的新系统上线,还需要批准,并且我们表明所做的每个更改都已经过测试(通过包含所有更改的快照)在此版本的生产请求中进行了测试)。

我们保留以前的生产版本以供出现问题时使用。

如果事情确实发生了问题(通过适当的测试程序和托管版本,他们不应该经常这样做),那么我们可以回滚或修复。 通常,当现场出现问题并且修复很小时,我们可以进行热修复,然后将修复移至测试以进行适当的测试。

无论如何,有时事情会过去...

New production releases go via our systems guys, the programmers and developers can only request to make their new system go live, approval is needed as well, and we show that each change that has been made has been tested (by including a snapshot of all that was tested in this release in the production request).

We keep the previous production releases for fallback in case of issues.

If things do break (which they shouldn't do often with a proper testing procedure and managed releases) then we can either roll back, or hotfix. Often when things are broken in live and the fix is small, we can hotfix, then move the fix to test to do a proper test.

Regardless, sometimes things get by...

野心澎湃 2024-07-30 10:54:53

只允许某些帐户写入访问权限,因此您必须以不同的方式登录才能

在Web服务器上进行更改,有两种目录结构,相互镜像,一种只有一个ID可以写入,另一种是每个人都可以写入的暂存目录。

在数据库服务器上,有一个生产数据库,其中只有一个 ID 可以写入,有一个临时数据库,每个人都可以写入。 临时数据库可以将夜间备份恢复到其中。

但是,如果您的查询错误或临时系统中的某些资源占用过多,则将从生产中提取资源,并且机器可能会挂起。

only allow certain accounts write access, so you have to log in differently to make a change

on web server, have two directory structures, that mirror each other, one where only one ID can write, the other staging dir, everyone can write.

on database server, have one production db, where only one ID can write, have a staging db where everyone can write. the staging DB can have nightly backup restored to it.

HOWEVER, if you have a bad query or some resource hog in your staging system resources will be pulled from production, and the machine could hang.

白首有我共你 2024-07-30 10:54:53

对于 Web 和应用程序服务器,我会尝试将环境复制到新位置(但在同一环境中),并让受影响的人在副本上重现行为。 这至少可以让你在一定程度上避免意外与 100% 的客户发生关系。

对于数据库服务器,我会在生产系统上配置用户帐户以赋予他们只读权限。

For Web and Application Servers, I would try to copy the environment to a new location (but on the same environment) and have the affected people reproduce behavior on the copy. This will at least give you a level of separation from accidentally screwing with 100% of your clients.

For Database Servers, I would configure user accounts on the production system to give them read only rights.

擦肩而过的背影 2024-07-30 10:54:53

只读/来宾帐户。 严重地。 这与您不总是以 root 或管理员身份登录的原因相同。

Read-Only/Guest accounts. Seriously. It's the same reason you don't always login as root or Administrator.

爺獨霸怡葒院 2024-07-30 10:54:53

这是一件困难的事情,并且它符合“无暂存环境”的领域。

出于多种原因,最好有一个专用的(重复的)PROD,您可以使用它来分阶段部署……并进行调试,但我知道有时当您开始时,效果不会那么快或彻底如我们所愿。

我看到的一件事是使用虚拟机:除了调试环境之外,您还可以在虚拟机中创建一个迷你产品并使用它来调试。 考虑到您正在开发的应用程序类型,这可能不切实际,因此该领域的其他详细信息会有所帮助。

至于避免在调试期间更改 PROD:是否有理由需要更改任何内容以方便调试? 如果是这样,可能值得考虑另一种解决方法。

This is a tough thing, and it goes with the territory of "no staging environment."

For many reasons, it's best to have a dedicated (duplicate) of PROD you can use to stage deploys to...and to debug on, but I know that sometimes when you're starting out that doesn't work out as quickly or thoroughly as we'd want.

One thing I've seen work is the use of VMs: aside from the debug environment, you can create a mini-PROD in a VM and use that to debug. This may not be practical given the type of app you're developing, so additional detail in that area would be helpful.

As for avoiding changes to PROD during debugging: is there a reason you'd need to change anything to facilitate debugging? If so, that might be worth looking into solving another way.

疯狂的代价 2024-07-30 10:54:53

版本控制对于控制生产环境的更改非常有帮助 - 只需使您的生产环境成为存储库中适当目录的工作副本即可。 当您推出更新时,您的源代码控制系统会确保复制所有更改的文件。 当更新破坏某些内容时,您可以将生产工作副本回滚到未破坏的最后一个版本。 此外,您可以通过标签而不是后备箱检查您的生产厕所; 这样您就可以通过调整标签来决​​定将哪些存储库修订版应用于生产环境。

如果您不熟悉版本控制系统的概念,我建议您做一些研究。 它们在概念上很复杂,但非常有用和强大。 维基百科文章是一个很好的起点:
http://en.wikipedia.org/wiki/Revision_control

Version control is immensely helpful for controlling changes to production environments - just make your production environment a working copy of the appropriate directory or directories from the repository. When you roll out an update, your source control system makes sure that ALL the changed files get copied. When an update breaks something, you can roll the production working copy back to the last revision which wasn't broken. Also, you can check your production WC out from a tag instead of from the trunk; that way you can decide which repository revisions to apply to the production environment by adjusting the tag.

If you're not familiar with the concepts of version control systems, I'd advise you to do some research. They're conceptually complex but incredibly useful and powerful. The Wikipedia article is a good place to start:
http://en.wikipedia.org/wiki/Revision_control

烟织青萝梦 2024-07-30 10:54:53

抱歉,你必须有一个暂存环境。 这是无法回避的。 如果这意味着您必须剔除数据集的大小,那么这就是您必须做的。 如果您有 VMware 和 VMware 转换器,请在停机期间导入生产系统(这是一个耗时数小时的过程,因此可能不切实际)。

如果没有对生产数据库(或副本)的完全访问权限,您将无法解决某些类别的问题,性能就是其中之一。 但您确实应该构建一个临时环境,即使它位于某人的台式机上并具有精简的数据集。

除此之外,我过去不得不与其中的一些一起生活,实际上,除了大量备份之外,你无能为力。 您所做的每项更改都应该先进行增量备份。 这样,如果你搞错了某件事,你损失的金额就不会很大。 SQL Server 可以进行差异备份,从而限制用于备份的磁盘空间量。 甲骨文也可以。

I'm sorry, you have to have a staging environment. There's no getting around this. If it means you have to cull the size of your datasets, then that's what you have to do. Use VMware and VMware converter to import the production systems during down-periods, if you have them (this is a many-hour process, so maybe not practical).

There are a certain class of problems you can't solve without having full access to production DBs (or a copy), performance is one of these. But you really should build a staging environment, even if it's on someone's desktop machine with a stripped down dataset.

That aside, I've had to live my life with a few of these in the past, and really, there's nothing you can do except lots of backups. Every change you make should be preceded by incremental backups. That way if you fubar'd something, the amount you've lost is not substantial. SQL server can take differential backups that limit amount of diskspace used for backups. Oracle can as well.

只有影子陪我不离不弃 2024-07-30 10:54:53

如果您确实没有其他选择,并且这可能是一种长期情况...请考虑向应用程序数据(文件或数据库)添加某种方式,以将一组数据标记为“请上帝实际上不要主动更改”与此数据的生产状态相结合,当激活此标志时,结合流程中关键位置的数据转储,您可以在不实际操作数据的情况下执行大部分生产逻辑。

In case you really have no other choice, and it is likely to be a chronic situation... consider adding some way to the application data (files, or database) to flag a set of data as 'please god do not actually actively change production state with this data', combined with data dumps at critical positions in a process when this flag is activated, you may be able to exercise most of the production logic without the data actually being acted upon.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文