实现 C# 语言扩展
使用诸如Parallel Linq之类的系统,可以拆分匿名的执行单个机器内跨多个核心和线程的函数、查询等。我希望能够使用标准语言结构(如 for 循环(如 Parallel.For())、值类型(如 ints))扩展它以在多台机器上运行、struct
等,并将应用程序源代码修改保持在最低限度。理想情况下,这将允许我打开一个项目,向方法添加属性,然后重新编译以获得对增强功能的访问。
看来我需要以下内容:
能够捕获已编译的代码块(例如 lambda)并将其传递给在另一个节点上运行的工作进程以及所需的任何数据,或
提供一个预处理器捕获有问题的代码,按排序编译它模板项目将用对处理网络通信、缓存和访问任何其他所需资源的类的引用来替换变量引用等,并将生成的 DLL 发送到在其他计算机上运行的任何可用工作节点。
罗斯林似乎提供了一些在这里有用的工具。有没有办法连接到当前的编译管道以允许这样做?
编辑
好的,我知道这是可能的,因为这些伙计们做到了。问题是,如何?
Using systems such as Parallel Linq, it's possible to split up execution of anonymous functions, queries, etc across multiple cores and threads within a single machine. I'd like the ability to extend this to run across multiple machines using standard language constructs such as for loops (like Parallel.For()
), value types like int
s, struct
s, etc., and keep the application source modifications to a minimum. Ideally, this would allow me to open a project, add an attribute to a method, and recompile to gain access to the enhanced functionality.
It seems that I'd need something along the lines of:
The ability to capture a compiled block of code (such as a lambda) and pass it to a worker process running on another node, along with any data that is required, or
Provide a preprocessor that would capture the code in question, compile it in a sort of template project that would replace variable references, etc, with references to a class that would handle network communication, caching and access to any other required assets, and send the resulting DLL to any available worker nodes running on other machines.
Roslyn appears to provide some tools that would be useful here. Is there way to hook into the current compilation pipeline to allow this?
Edit
Okay, I know this is possible, because these guys did it. The question is, how?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(2)
听起来很棒。事实上,我们有一个非常类似于微软研究院的系统,尽管显然我无法讨论细节。
好的,您已经得到了。我们将该功能添加到 C# 3 中。这就是 LINQ to SQL 的工作原理。 不知何故 LINQ 查询必须进入数据库。编译后的 lambda 在客户端计算机上进行询问,转换为发送到服务器节点的查询,然后将结果发回。
这不是罗斯林的目的;这是他的目的。 Roslyn 并不是要向 C# 语言添加新功能。它是为了更容易地分析代码以构建重构引擎之类的东西。
您不需要连接到编译管道。 PLINQ 不会更改编译器,LINQ to SQL 不会更改编译器,等等。当您将 lambda 转换为表达式树时,编译器会发出代码,在运行时创建表示 lambda 的表达式树。您可以询问该表达式树,将其序列化到网络中的另一台计算机,反序列化它,将其转换为委托并运行它(如果您喜欢这样做)。
您可能需要编写自己的表达式树序列化器和反序列化器,但它们是非常简单的数据结构。作为一棵不可变的树应该使它们很容易序列化和反序列化;它们不能真正形成复杂的网络,因为它们总是从叶节点向上构建。
Sounds great. In fact we have a system very much like that over in Microsoft Research, though obviously I cannot discuss the details.
OK, you've got it. We added that feature to C# 3. That's how LINQ to SQL works. Somehow the LINQ query has to get onto the database. The compiled lambda is interrogated on the client machine, transformed into a query which is sent to the server node, and then the result is sent back.
That's not the purpose of Roslyn; Roslyn is not about adding new features to the C# language. It's about making it easier to analyze code to build things like refactoring engines.
You don't need to hook into the compilation pipeline. PLINQ doesn't change the compiler, LINQ to SQL doesn't change the compiler, and so on. When you convert a lambda to an expression tree the compiler emits code that creates an expression tree at runtime that represents the lambda. You can interrogate that expression tree, serialize it across to another machine in your network, deserialize it, turn it into a delegate and run it if that's the kind of thing you enjoy doing.
You'll need to write your own expression tree serializer and deserializer probably, but they are pretty straightforward data structures. Being an immutable tree should make them pretty easy to serialize and deserialize; they can't really form complex networks since they are always constructed from leaf nodes up.
你不必扩展语言 persay 来完成 Brahma 所做的事情。他刚刚实现了一个自定义查询提供程序,用于解析表达式树并发出 GPGPU 代码(LINQ to SQL 执行相同的操作,但使用 SQL)。
我在 MSDN 此处链接了一个基本指南,可以帮助您启动并运行实现一个
IQueryable
提供者。困难的部分是遍历表达式树并生成 OpenCL 代码。一旦你能做到这一点,你只需将其传递给 Cloo,然后你就应该运行了。
编辑
您使用属性
[Kernel]
链接了一个将标准.NET代码编译为GPU代码的工具。他们通过使用构建后工具查找编译后的 IL 中的属性并执行 IL 重写来生成 GPU 调用来实现此目的。这类似于 AOP 解决方案 PostSharp。IL 重写既费时又费力,但您也可以走这条路。
You don't have to extend the language persay to do what Brahma does. He just implemented a custom query provider that parses expression trees and emits GPGPU code (LINQ to SQL does the same thing but with SQL).
I linked a basic guide on MSDN here that can get you up and running implementing an
IQueryable
provider.The hard part will be traversing the expression trees and generating OpenCL code. Once you can do that you just pass it off to Cloo and you should be running.
Edit
You linked a tool that compiles standard .NET code to GPU code with an attribute
[Kernel]
. They do this by having a post-build tool look for the attributes in the compiled IL and they perform IL rewriting to generate GPU calls. This is similar to PostSharp, an AOP solution.IL rewriting is time consuming and hard work but you could also go this route.